Re: [PR] [PostgreSQL] Add DeserializationConverterFactory for PostGIS schemas [flink-cdc]
yuxiqian commented on PR #1161: URL: https://github.com/apache/flink-cdc/pull/1161#issuecomment-2076487495 Hi @Elephantusparvus, thanks for your contribution! Seems this issue has been addressed by #1554, would you mind if I close this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][docs] Fix the invalid link in 1.18 release note [flink]
reswqa commented on PR #24713: URL: https://github.com/apache/flink/pull/24713#issuecomment-2076484999 Thanks @yebukong. I will merge this after AZP green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35228][Connectors/Kafka] Fix DynamicKafkaSourceEnumerator removed topics state cleanup [flink-connector-kafka]
IgnasD commented on PR #97: URL: https://github.com/apache/flink-connector-kafka/pull/97#issuecomment-2076484735 The fix is that [previously](https://github.com/apache/flink-connector-kafka/blob/369e7be46a70fd50d68746498aed82105741e7d6/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumerator.java#L291C39-L291C60) `activeTopicPartitions` were being constructed but used nowhere. Now I'm constructing `KafkaSourceEnumState` and passing `activeTopicPartitions` to it. And yes, while doing that some refactoring came in naturally to avoid code duplication. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] expose meta_ts field in the metadata through flink sql [flink-cdc]
yuxiqian commented on PR #1141: URL: https://github.com/apache/flink-cdc/pull/1141#issuecomment-2076484397 Hi @EMsnap, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit, and address @ruanhang1993's comments? Kindly reminder that `com.ververica.cdc.connectors.mysql` package has been moved to `org.apache.flink.cdc.connectors.mysql`. Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34980] Translate overview document into Chinese [flink-kubernetes-operator]
RocMarshal commented on code in PR #810: URL: https://github.com/apache/flink-kubernetes-operator/pull/810#discussion_r1578934746 ## docs/content.zh/docs/concepts/overview.md: ## @@ -104,6 +122,7 @@ drwxr-xr-x 2 60 May 12 09:46 a6031ec7-ab3e-4b30-ba77-6498e58e6b7f drwxr-xr-x 2 60 May 11 15:11 b6fb2a9c-d1cd-4e65-a9a1-e825c4b47543 ``` -### AuditUtils can log sensitive information present in the custom resources -As reported in [FLINK-30306](https://issues.apache.org/jira/browse/FLINK-30306) when Flink custom resources change the operator logs the change, which could include sensitive information. We suggest ingesting secrets to Flink containers during runtime to mitigate this. -Also note that anyone who has access to the custom resources already had access to the potentially sensitive information in question, but folks who only have access to the logs could also see them now. We are planning to introduce redaction rules to AuditUtils to improve this in a later release. + Review Comment: ```suggestion ``` We should keep the `-` style at the link anchors in the whole page. BTW:I am not sure if it is necessary to lowercase all characters, which needs to be consistent with the English title format. Would you mind checking and making adjustments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] expose update_before field in the metadata through flink sql [flink-cdc]
yuxiqian commented on PR #1139: URL: https://github.com/apache/flink-cdc/pull/1139#issuecomment-2076482569 Hi @EMsnap, sorry for the delay of this PR. Seems this issue has been resolved by migrating to `ROW_KIND` structure with `UPDATE_BEFORE` support. Would you mind if I close this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34980] Translate overview document into Chinese [flink-kubernetes-operator]
RocMarshal commented on code in PR #810:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/810#discussion_r1578923881
##
docs/content.zh/docs/concepts/overview.md:
##
@@ -24,78 +24,96 @@ specific language governing permissions and limitations
under the License.
-->
-# Overview
-Flink Kubernetes Operator acts as a control plane to manage the complete
deployment lifecycle of Apache Flink applications. Although Flink’s native
Kubernetes integration already allows you to directly deploy Flink applications
on a running Kubernetes(k8s) cluster, [custom
resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
and the [operator
pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) have
also become central to a Kubernetes native deployment experience.
-
-Flink Kubernetes Operator aims to capture the responsibilities of a human
operator who is managing Flink deployments. Human operators have deep knowledge
of how Flink deployments ought to behave, how to start clusters, how to deploy
jobs, how to upgrade them and how to react if there are problems. The main goal
of the operator is the automation of these activities, which cannot be achieved
through the Flink native integration alone.
-
-## Features
-### Core
-- Fully-automated [Job Lifecycle Management]({{< ref
"docs/custom-resource/job-management" >}})
- - Running, suspending and deleting applications
- - Stateful and stateless application upgrades
- - Triggering and managing savepoints
- - Handling errors, rolling-back broken upgrades
+
+
+# 概述
+Flink Kubernetes Operator 扮演控制平面的角色，用于管理 Apache Flink 应用程序的完整部署生命周期。尽管 Flink
的原生 Kubernetes 集成已经允许你直接在运行的 Kubernetes(k8s) 集群上部署 Flink 应用程序，但
[自定义资源](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
和 [operator
模式](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) 也已成为
Kubernetes 本地部署体验的核心。
+
+Flink Kubernetes Operator 旨在承担管理 Flink deployments , 这是 human Operator 的职责。
human Operators 对 Flink 部署应该如何运行、如何启动集群、如何部署作业、如何升级作业以及出现问题时如何反应有着深入的了解。
Operator 的主要目标是这些活动的自动化，这无法仅通过 Flink 原生集成来实现。
+
+
+
+
+## 特征
+
+
+
+### 核心
+- 全自动 [Job Lifecycle Management]({{< ref "docs/custom-resource/job-management"
>}})
+ - 运行、暂停和删除应用程序
+ - 有状态和无状态应用程序升级
+ - 保存点的触发和管理
+ - 任务管理器的扩展和缩减
- Multiple Flink version support: v1.15, v1.16, v1.17, v1.18
Review Comment:
多 Flink 版本支持：v1.15，v1.16，v1.17，v1.18
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [mysql] expose op type in the metadata through flink sql [flink-cdc]
yuxiqian commented on PR #1137: URL: https://github.com/apache/flink-cdc/pull/1137#issuecomment-2076479642 Hi @EMsnap, sorry for the delay of this PR. Seems this issue has been addressed in https://github.com/apache/flink-cdc/pull/2927. Would you mind if I close this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32622][table-planner] Optimize mini-batch assignment [flink]
xuyangzhong commented on code in PR #23470:
URL: https://github.com/apache/flink/pull/23470#discussion_r1578935653
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/optimize/StreamCommonSubGraphBasedOptimizer.scala:
##
@@ -46,11 +47,14 @@ import scala.collection.JavaConversions._
class StreamCommonSubGraphBasedOptimizer(planner: StreamPlanner)
extends CommonSubGraphBasedOptimizer {
- override protected def doOptimize(roots: Seq[RelNode]): Seq[RelNodeBlock] = {
-val tableConfig = planner.getTableConfig
-// build RelNodeBlock plan
-val sinkBlocks = RelNodeBlockPlanBuilder.buildRelNodeBlockPlan(roots,
tableConfig)
-// infer trait properties for sink block
+ private def optimizeSinkBlocks(
+ origMiniBatchEnabled: Boolean,
+ tableConfig: TableConfig,
+ sinkBlocks: Seq[RelNodeBlock]): Seq[RelNodeBlock] = {
+if (origMiniBatchEnabled)
Review Comment:
nit: How about moving this if block from function `optimizeSinkBlocks` into
`doOptimize`? Because confining all minibatch-related concepts within one
function seems cleaner.
Just like:
```
override protected def doOptimize(roots: Seq[RelNode]): Seq[RelNodeBlock] = {
val tableConfig = planner.getTableConfig
// build RelNodeBlock plan
val sinkBlocks = RelNodeBlockPlanBuilder.buildRelNodeBlockPlan(roots,
tableConfig)
// infer trait properties for sink block
val origMiniBatchEnabled =
tableConfig.get(ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED)
try {
if (origMiniBatchEnabled) {
tableConfig.set(
ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED,
Boolean.box(!shouldSkipMiniBatch(sinkBlocks)))
}
optimizeSinkBlocks(tableConfig, sinkBlocks)
} finally {
tableConfig.getConfiguration.set(
ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED,
origMiniBatchEnabled)
}
}
private def optimizeSinkBlocks(
tableConfig: TableConfig,
sinkBlocks: Seq[RelNodeBlock]): Seq[RelNodeBlock] = {
sinkBlocks.foreach {
sinkBlock =>
..
```
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/optimize/StreamCommonSubGraphBasedOptimizer.scala:
##
@@ -83,25 +87,42 @@ class StreamCommonSubGraphBasedOptimizer(planner:
StreamPlanner)
isSinkBlock = true)
block.setOptimizedPlan(optimizedTree)
return sinkBlocks
+} else {
Review Comment:
nit: It seems that reverting this part of the changes won't be an issue. How
about reverting it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [mysql] keep order of each split by replacing HashMap with LinkedHashMap [flink-cdc]
yuxiqian commented on PR #1075: URL: https://github.com/apache/flink-cdc/pull/1075#issuecomment-2076469504 Hi @yujiaxinlong, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors.mysql` package has been moved to `org.apache.flink.cdc.connectors.mysql`. Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35223][rest] Add jobType in JobDetailsInfo related rest api [flink]
flinkbot commented on PR #24718: URL: https://github.com/apache/flink/pull/24718#issuecomment-2076466938 ## CI report: * 68eac76bf3ae406bb4757c3d3d450e240d76a4cb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34980] Translate overview document into Chinese [flink-kubernetes-operator]
RocMarshal commented on code in PR #810:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/810#discussion_r1574398142
##
docs/content.zh/docs/concepts/overview.md:
##
@@ -24,86 +24,93 @@ specific language governing permissions and limitations
under the License.
-->
-# Overview
-Flink Kubernetes Operator acts as a control plane to manage the complete
deployment lifecycle of Apache Flink applications. Although Flink’s native
Kubernetes integration already allows you to directly deploy Flink applications
on a running Kubernetes(k8s) cluster, [custom
resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
and the [operator
pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) have
also become central to a Kubernetes native deployment experience.
+
-Flink Kubernetes Operator aims to capture the responsibilities of a human
operator who is managing Flink deployments. Human operators have deep knowledge
of how Flink deployments ought to behave, how to start clusters, how to deploy
jobs, how to upgrade them and how to react if there are problems. The main goal
of the operator is the automation of these activities, which cannot be achieved
through the Flink native integration alone.
Review Comment:
ignored ?
##
docs/content.zh/docs/concepts/overview.md:
##
@@ -24,78 +24,96 @@ specific language governing permissions and limitations
under the License.
-->
-# Overview
-Flink Kubernetes Operator acts as a control plane to manage the complete
deployment lifecycle of Apache Flink applications. Although Flink’s native
Kubernetes integration already allows you to directly deploy Flink applications
on a running Kubernetes(k8s) cluster, [custom
resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
and the [operator
pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) have
also become central to a Kubernetes native deployment experience.
-
-Flink Kubernetes Operator aims to capture the responsibilities of a human
operator who is managing Flink deployments. Human operators have deep knowledge
of how Flink deployments ought to behave, how to start clusters, how to deploy
jobs, how to upgrade them and how to react if there are problems. The main goal
of the operator is the automation of these activities, which cannot be achieved
through the Flink native integration alone.
-
-## Features
-### Core
-- Fully-automated [Job Lifecycle Management]({{< ref
"docs/custom-resource/job-management" >}})
- - Running, suspending and deleting applications
- - Stateful and stateless application upgrades
- - Triggering and managing savepoints
- - Handling errors, rolling-back broken upgrades
+
+
+# 概述
+Flink Kubernetes Operator 扮演控制平面的角色，用于管理 Apache Flink 应用程序的完整部署生命周期。尽管 Flink
的原生 Kubernetes 集成已经允许你直接在运行的 Kubernetes(k8s) 集群上部署 Flink 应用程序，但
[自定义资源](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
和 [operator
模式](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) 也已成为
Kubernetes 本地部署体验的核心。
+
+Flink Kubernetes Operator 旨在承担管理 Flink deployments , 这是 human Operator 的职责。
human Operators 对 Flink 部署应该如何运行、如何启动集群、如何部署作业、如何升级作业以及出现问题时如何反应有着深入的了解。
Operator 的主要目标是这些活动的自动化，这无法仅通过 Flink 原生集成来实现。
Review Comment:
```suggestion
Flink Kubernetes Operator 旨在承担管理 Flink deployments，这是 human Operator
的职责。human Operators 对 Flink
部署应该如何运行、如何启动集群、如何部署作业、如何升级作业以及出现问题时如何反应有着深入的了解。Operator 的主要目标是这些活动的自动化，而这无法仅通过
Flink 原生集成来实现。
```
##
docs/content.zh/docs/concepts/overview.md:
##
@@ -24,78 +24,96 @@ specific language governing permissions and limitations
under the License.
-->
-# Overview
-Flink Kubernetes Operator acts as a control plane to manage the complete
deployment lifecycle of Apache Flink applications. Although Flink’s native
Kubernetes integration already allows you to directly deploy Flink applications
on a running Kubernetes(k8s) cluster, [custom
resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
and the [operator
pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) have
also become central to a Kubernetes native deployment experience.
-
-Flink Kubernetes Operator aims to capture the responsibilities of a human
operator who is managing Flink deployments. Human operators have deep knowledge
of how Flink deployments ought to behave, how to start clusters, how to deploy
jobs, how to upgrade them and how to react if there are problems. The main goal
of the operator is the automation of these activities, which cannot be achieved
through the Flink native integration alone.
-
-## Features
-### Core
-- Fully-automated [Job Lifecycle Management]({{< ref
"docs/custom-resource/job-management" >}})
- - Running, suspending and deleting applications
- - Stateful and stateless applica
[jira] [Updated] (FLINK-35223) Add jobType in JobDetailsInfo related rest api
[ https://issues.apache.org/jira/browse/FLINK-35223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35223: --- Labels: pull-request-available (was: ) > Add jobType in JobDetailsInfo related rest api > -- > > Key: FLINK-35223 > URL: https://issues.apache.org/jira/browse/FLINK-35223 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] support op_type metadata columns [flink-cdc]
yuxiqian commented on PR #1071: URL: https://github.com/apache/flink-cdc/pull/1071#issuecomment-2076455888 Hi @wangqinghuan, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors` package has been moved to `org.apache.flink.cdc.connectors`. Also, there's been quite a few new connectors since your commit. Maybe we should add them along for consistency. WDYT? Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] support for transaction metadata events [flink-cdc]
yuxiqian commented on PR #1039: URL: https://github.com/apache/flink-cdc/pull/1039#issuecomment-2076450755 Hi @rashminpatel123, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors.mysql` package has been moved to `org.apache.flink.cdc.connectors.mysql`. Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][docs] Fix the invalid link in 1.18 release note [flink]
yebukong commented on PR #24713: URL: https://github.com/apache/flink/pull/24713#issuecomment-2076450701 @reswqa I have already changed the commit message. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added option for MysqlSource to dispatch watermark event at the begin… [flink-cdc]
yuxiqian commented on PR #883: URL: https://github.com/apache/flink-cdc/pull/883#issuecomment-2076447317 Hi @qidian99, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that anything under `com.ververica.cdc.connectors.mysql` package has been moved to `org.apache.flink.cdc.connectors.mysql`. Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35156][Runtime] Make operators of DataStream V2 integrate with async state processing framework [flink]
fredia merged PR #24678: URL: https://github.com/apache/flink/pull/24678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-3.1][FLINK-35128][cdc-connector][cdc-base] Re-calculate the starting changelog offset after the new table added (#3230) [flink-cdc]
loserwang1024 closed pull request #3256: [BP-3.1][FLINK-35128][cdc-connector][cdc-base] Re-calculate the starting changelog offset after the new table added (#3230) URL: https://github.com/apache/flink-cdc/pull/3256 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] expose Enumerator metrics via Source Event [flink-cdc]
yuxiqian commented on PR #853: URL: https://github.com/apache/flink-cdc/pull/853#issuecomment-2076438387 Hi @zhjwpku, sorry for the long delay of this PR. Could you please help rebase it with latest master branch before we can merge this? A complete refactor had been made since your original commit, from `com.alibaba.ververica.cdc.connectors.mysql.source` to `org.apache.flink.cdc.connectors.mysql.source`. Thanks again for your generous contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35128) Re-calculate the starting change log offset after the new table added
[ https://issues.apache.org/jira/browse/FLINK-35128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840657#comment-17840657 ] Qingsheng Ren commented on FLINK-35128: --- flink-cdc master: bdca0e328bce0aa2dc05153bc67da4c4875586fd > Re-calculate the starting change log offset after the new table added > - > > Key: FLINK-35128 > URL: https://issues.apache.org/jira/browse/FLINK-35128 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Hongshun Wang >Assignee: Hongshun Wang >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > > In mysql cdc, re-calculate the starting binlog offset after the new table > added in MySqlBinlogSplit#appendFinishedSplitInfos, while there lack of same > action in StreamSplit#appendFinishedSplitInfos. This will cause data loss if > any newly added table snapshot split's highwatermark is smaller. > > Some unstable test problem occurs because of it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35128][cdc-connector][cdc-base] Re-calculate the starting changelog offset after the new table added [flink-cdc]
PatrickRen merged PR #3230: URL: https://github.com/apache/flink-cdc/pull/3230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35234) Fix NullPointerException of org.apache.flink.cdc.common.configuration.ConfigurationUtils#convertToString
[
https://issues.apache.org/jira/browse/FLINK-35234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35234:
---
Labels: pull-request-available (was: )
> Fix NullPointerException of
> org.apache.flink.cdc.common.configuration.ConfigurationUtils#convertToString
>
>
> Key: FLINK-35234
> URL: https://issues.apache.org/jira/browse/FLINK-35234
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.1.0
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: cdc-3.2.0
>
>
> Exception like this:
> {code:java}
> Caused by: java.lang.NullPointerExceptionat
> org.apache.flink.cdc.common.configuration.ConfigurationUtils.convertToString(ConfigurationUtils.java:133)
> ~[?:?]at
> org.apache.flink.cdc.common.configuration.Configuration.toMap(Configuration.java:138)
> ~[?:?] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35234][hotfix][cdc-common] Fix NullPointerException of org.apache.flink.cdc.common.configuration.ConfigurationUtils#convertToString [flink-cdc]
loserwang1024 commented on PR #3255: URL: https://github.com/apache/flink-cdc/pull/3255#issuecomment-2076431237 @yuxiqian , @PatrickRen , a mirror fix, CC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-35234][hotfix][cdc-common] Fix NullPointerException of org.apache.flink.cdc.common.configuration.ConfigurationUtils#convertToString [flink-cdc]
loserwang1024 opened a new pull request, #3255: URL: https://github.com/apache/flink-cdc/pull/3255 Fix exception like this: ```java Caused by: java.lang.NullPointerExceptionat org.apache.flink.cdc.common.configuration.ConfigurationUtils.convertToString(ConfigurationUtils.java:133) ~[?:?]at org.apache.flink.cdc.common.configuration.Configuration.toMap(Configuration.java:138) ~[?:?] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix: fix mysql-cdc case sensitive [flink-cdc]
yuxiqian commented on PR #831: URL: https://github.com/apache/flink-cdc/pull/831#issuecomment-2076427913 Hello @shizhengchao, sorry for the long delay of this PR. Could you please help rebase it with latest master branch before we can merge this? Kindly reminder about some refactored packages: * `MySQLTableSource` and `MySQLTableSourceFactory` `com.ververica.cdc.connectors.mysql.table` → `org.apache.flink.cdc.connectors.mysql.table` Thanks again for your generous contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35234) Fix NullPointerException of org.apache.flink.cdc.common.configuration.ConfigurationUtils#convertToString
Hongshun Wang created FLINK-35234:
-
Summary: Fix NullPointerException of
org.apache.flink.cdc.common.configuration.ConfigurationUtils#convertToString
Key: FLINK-35234
URL: https://issues.apache.org/jira/browse/FLINK-35234
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: Hongshun Wang
Fix For: cdc-3.2.0
Exception like this:
{code:java}
Caused by: java.lang.NullPointerExceptionat
org.apache.flink.cdc.common.configuration.ConfigurationUtils.convertToString(ConfigurationUtils.java:133)
~[?:?]at
org.apache.flink.cdc.common.configuration.Configuration.toMap(Configuration.java:138)
~[?:?] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35220) the error of mysql cdc
[
https://issues.apache.org/jira/browse/FLINK-35220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840656#comment-17840656
]
Hongshun Wang commented on FLINK-35220:
---
See
https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/faq/faq/#q13-the-job-reports-an-error-eventdatadeserializationexception-failed-to-deserialize-data-of-eventheaderv4--caused-by-javanetsocketexception-connection-resetwhat-should-i-do
> the error of mysql cdc
> ---
>
> Key: FLINK-35220
> URL: https://issues.apache.org/jira/browse/FLINK-35220
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Reporter: xiaotouming
>Priority: Blocker
>
> When we listen for incremental data on a table,there is a error occur once。
> the log is:
> 2024-04-15 04:52:36
> java.lang.RuntimeException: One or more fetchers have encountered exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:261)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:131)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:157)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received
> unexpected exception while polling the records
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:114)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> Caused by:
> com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.errors.ConnectException:
> An exception occurred in the change event producer. This connector will be
> stopped.
> at
> io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
> at
> com.ververica.cdc.connectors.mysql.debezium.task.context.MySqlErrorHandler.setProducerThrowable(MySqlErrorHandler.java:85)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1544)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1079)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
> ... 1 more
> Caused by: io.debezium.DebeziumException: Failed to deserialize data of
> EventHeaderV4\{timestamp=171312635, eventType=EXT_UPDATE_ROWS,
> serverId=1655776775, headerLength=19, dataLength=3291,
> nextPosition=647993741, flags=0}
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1488)
> ... 5 more
> Caused by:
> com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException:
> Failed to deserialize data of EventHeaderV4\{timestamp=171312635,
> eventType=EXT_UPDATE_ROWS, serverId=1655776775, headerLength=19,
> dataLength=3291, nextPosition=647993741, flags=0}
> at
> com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:341)
> at
> co
Re: [PR] [mysql] Add schema validation for table schema #493 [flink-cdc]
yuxiqian commented on PR #522: URL: https://github.com/apache/flink-cdc/pull/522#issuecomment-2076420461 Hi @luoyuxia, thanks for your contribution! Could you please rebase this PR with latest `master` branch before we can merge it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] optimize the update operation in binlog events [flink-cdc]
yuxiqian commented on PR #41: URL: https://github.com/apache/flink-cdc/pull/41#issuecomment-2076415796 Hello @shizhengchao, sorry for the long delay of this PR. Could you please help rebase it with latest `master` branch before we can merge this? Kindly reminder about some refactored packages: * `RowDataDebeziumDeserializeSchema` `com.alibaba.ververica.cdc.debezium.table` → `org.apache.flink.cdc.debezium.table` * `MySQLTableSource` and `MySQLTableSourceFactory` `com.alibaba.ververica.cdc.connectors.mysql.table` → `org.apache.flink.cdc.connectors.mysql.table` * `PostgreSQLTableSource` and `PostgreSQLTableFactory` `com.alibaba.ververica.cdc.connectors.postgres.table` → `org.apache.flink.cdc.connectors.postgres.table` Also, there are a lot of new connectors added into Flink CDC since this PR was submitted. Maybe we can bring the `capture-unchanged-updates` option to more connectors? Thanks again for your generous contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-35097) Table API Filesystem connector with 'raw' format repeats last line
[
https://issues.apache.org/jira/browse/FLINK-35097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-35097.
Fix Version/s: 1.18.2
1.20.0
1.19.1
Resolution: Fixed
Fixed in master:
3a56c2f7796557328ef33b6caae57bca5c18
8ffe6d698fb4f37c68a3914e3903dc874dfcb999
51d015b570497c57f56bea64371ecec23fd454c1
Fixed in 1.19:
e7816f714ef5298e1ca978aeddf62732794bb93f
Fixed in 1.18:
cf216f03c71433d21af39d125b4ff9d804cb0b38
> Table API Filesystem connector with 'raw' format repeats last line
> --
>
> Key: FLINK-35097
> URL: https://issues.apache.org/jira/browse/FLINK-35097
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.17.1
> Environment: I ran the above test with 1.17.1. I checked for existing
> bug tickets and release notes, but did not find anything, so assuming this
> effects 1.18 and 1.19.
>Reporter: David Perkins
>Assignee: Kumar Mallikarjuna
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0, 1.19.1
>
>
> When using the Filesystem connector with 'raw' format to read text data that
> contains new lines, a row is returned for every line, but always contains the
> contents of the last line.
> For example, with the following file.
> {quote}
> line 1
> line 2
> line 3
> {quote}
> And table definition
> {quote}
> create TABLE MyRawTable (
> `doc` string,
> ) WITH (
> 'path' = 'file:///path/to/data',
> 'format' = 'raw',
>'connector' = 'filesystem'
> );
> {quote}
> Selecting `*` from the table produces three rows all with "line 3" for `doc`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-34902) INSERT INTO column mismatch leads to IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-34902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther reopened FLINK-34902:
--
> INSERT INTO column mismatch leads to IndexOutOfBoundsException
> --
>
> Key: FLINK-34902
> URL: https://issues.apache.org/jira/browse/FLINK-34902
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Timo Walther
>Assignee: Jeyhun Karimov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> SQL:
> {code}
> INSERT INTO t (a, b) SELECT 1;
> {code}
>
> Stack trace:
> {code}
> org.apache.flink.table.api.ValidationException: SQL validation failed. Index
> 1 out of bounds for length 1
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:200)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:117)
> at
> Caused by: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for
> length 1
> at
> java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
> at
> java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
> at
> java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
> at java.base/java.util.Objects.checkIndex(Objects.java:374)
> at java.base/java.util.ArrayList.get(ArrayList.java:459)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1(PreValidateReWriter.scala:355)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1$adapted(PreValidateReWriter.scala:355)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34902) INSERT INTO column mismatch leads to IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-34902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840650#comment-17840650
]
Timo Walther commented on FLINK-34902:
--
[~jeyhunkarimov] could you take another look at your changes? It seems this
broke statements including VALUES.
works:
{code}
INSERT INTO t_3_columns (id, name, num) SELECT 'id_0', 'name_0', 5;
{code}
doesn't work:
{code}
INSERT INTO t_3_columns (id, name, num) VALUES ('id_0', 'name_0', 5);
{code}
I will reopen this issue.
> INSERT INTO column mismatch leads to IndexOutOfBoundsException
> --
>
> Key: FLINK-34902
> URL: https://issues.apache.org/jira/browse/FLINK-34902
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Timo Walther
>Assignee: Jeyhun Karimov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> SQL:
> {code}
> INSERT INTO t (a, b) SELECT 1;
> {code}
>
> Stack trace:
> {code}
> org.apache.flink.table.api.ValidationException: SQL validation failed. Index
> 1 out of bounds for length 1
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:200)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:117)
> at
> Caused by: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for
> length 1
> at
> java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
> at
> java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
> at
> java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
> at java.base/java.util.Objects.checkIndex(Objects.java:374)
> at java.base/java.util.ArrayList.get(ArrayList.java:459)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1(PreValidateReWriter.scala:355)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1$adapted(PreValidateReWriter.scala:355)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [hotfix][docs] Fix the invalid link to FLINK-32468 [flink]
reswqa commented on PR #24713: URL: https://github.com/apache/flink/pull/24713#issuecomment-2076339397 Thanks for the fix. Could you change the commit message to `[hotfix][docs] Fix the invalid link in 1.18 release note`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35215][core] Fix the bug when Kryo serialize length is 0 [flink]
reswqa commented on PR #24717: URL: https://github.com/apache/flink/pull/24717#issuecomment-2076333034 Not to block this, just curious do you know why FLINK-34954 affects performance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
Zakelly commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1578825952
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -220,12 +283,17 @@ void insertBlockingBuffer(StateRequest request) {
* @param force whether to trigger requests in force.
*/
void triggerIfNeeded(boolean force) {
-// TODO: introduce a timeout mechanism for triggering.
if (!force && stateRequestsBuffer.activeQueueSize() < batchSize) {
+// if the active queue size is 1, it means that the current
request is the oldest one in
+// the active queue.
+if (stateRequestsBuffer.activeQueueSize() == 1) {
+scheduleTimeout(currentTriggerSeq.get());
+}
Review Comment:
I mean, the `triggerIfNeeded(false)` may be triggered somewhere else right?
And even the `enqueueToActive` has two entry. I'd suggest move the
`scheduleTimeout` part into the `StateRequestBuffer`, but the handler and seq
maintaining part should stay in `AEC`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [mysql] Support docker image of ARM chip [flink-cdc]
yuxiqian commented on PR #1368: URL: https://github.com/apache/flink-cdc/pull/1368#issuecomment-2076325048 Hi @minchowang, Thanks for your contribution! Now MySQL testcases have all been upgraded to 8.0 and they provide native ARM64 images already, so seems #1367 has been resolved. Would you mind if I close this PR or do you have any other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
fredia commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1578815645
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -220,12 +283,17 @@ void insertBlockingBuffer(StateRequest request) {
* @param force whether to trigger requests in force.
*/
void triggerIfNeeded(boolean force) {
-// TODO: introduce a timeout mechanism for triggering.
if (!force && stateRequestsBuffer.activeQueueSize() < batchSize) {
+// if the active queue size is 1, it means that the current
request is the oldest one in
+// the active queue.
+if (stateRequestsBuffer.activeQueueSize() == 1) {
+scheduleTimeout(currentTriggerSeq.get());
+}
Review Comment:
No, there will definitely be a trigger between two
`stateRequestsBuffer.activeQueueSize() == 1`, so when the two
`stateRequestsBuffer.activeQueueSize() == 1` conditions are met, seq number
must be different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [hotfix] make sure run RecordContext#release() in Task thread. [flink]
zoltar9264 commented on PR #24705: URL: https://github.com/apache/flink/pull/24705#issuecomment-2076289229 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] make sure run RecordContext#release() in Task thread. [flink]
zoltar9264 commented on PR #24705: URL: https://github.com/apache/flink/pull/24705#issuecomment-2076288986 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35190][table] Support create materialized table syntax [flink]
lsyldliu commented on code in PR #24707:
URL: https://github.com/apache/flink/pull/24707#discussion_r1578771973
##
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/ddl/SqlCreateMaterializedTable.java:
##
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.sql.parser.ddl;
+
+import org.apache.flink.sql.parser.SqlUnparseUtils;
+import org.apache.flink.sql.parser.ddl.constraint.SqlTableConstraint;
+
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlCreate;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlIntervalLiteral;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.util.ImmutableNullableList;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+import static java.util.Objects.requireNonNull;
+
+/** CREATE MATERIALIZED TABLE DDL sql call. */
+public class SqlCreateMaterializedTable extends SqlCreate {
+
+public static final SqlSpecialOperator OPERATOR =
+new SqlSpecialOperator("CREATE MATERIALIZED TABLE",
SqlKind.CREATE_TABLE);
+
+private final SqlIdentifier tableName;
+
+private final SqlCharStringLiteral comment;
+
+private final SqlTableConstraint tableConstraint;
+
+private final SqlNodeList partitionKeyList;
+
+private final SqlNodeList propertyList;
+
+// SqlIntervalLiteral
Review Comment:
Remove this comment, it makes no sense.
##
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/ddl/SqlCreateMaterializedTable.java:
##
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.sql.parser.ddl;
+
+import org.apache.flink.sql.parser.SqlUnparseUtils;
+import org.apache.flink.sql.parser.ddl.constraint.SqlTableConstraint;
+
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlCreate;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlIntervalLiteral;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.util.ImmutableNullableList;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+import static java.util.Objects.requireNonNull;
+
+/** CREATE MATERIALIZED TABLE DDL sql call. */
+public class SqlCreateMaterializedTable extends SqlCreate {
+
+public static final SqlSpecialOperator OPERATOR =
+new SqlSpecialOperator("CREATE MATERIALIZED TABLE",
SqlKind.CREATE_TABLE);
+
+private final SqlIdentifier tableName;
+
+private final SqlCharStringLiteral comment;
+
+private final SqlTableConstraint tableConstraint;
+
+private final SqlNodeList
Re: [PR] [FLINK-35233] Fix lookup cache reuse RowData object problem [flink-connector-hbase]
Tan-JiaLiang commented on PR #47: URL: https://github.com/apache/flink-connector-hbase/pull/47#issuecomment-2076275602 @ferenc-csaky Can you please take a look when you are free? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35215][core] Fix the bug when Kryo serialize length is 0 [flink]
flinkbot commented on PR #24717: URL: https://github.com/apache/flink/pull/24717#issuecomment-2076272429 ## CI report: * f09de068978ccc147fffc47925232b7bfeb1f584 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35233] Fix lookup cache reuse RowData object problem [flink-connector-hbase]
Tan-JiaLiang commented on PR #47: URL: https://github.com/apache/flink-connector-hbase/pull/47#issuecomment-2076271825 @MartijnVisser If you have some free time, can you help to trigger a CI run? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35215: --- Labels: pull-request-available (was: ) > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Priority: Blocker > Labels: pull-request-available > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > [http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35233) HBase lookup result is wrong when lookup cache is enabled
[
https://issues.apache.org/jira/browse/FLINK-35233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35233:
---
Labels: pull-request-available (was: )
> HBase lookup result is wrong when lookup cache is enabled
> -
>
> Key: FLINK-35233
> URL: https://issues.apache.org/jira/browse/FLINK-35233
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: hbase-3.0.0
>Reporter: tanjialiang
>Priority: Critical
> Labels: pull-request-available
>
> HBase table
> ||rowkey||name||age||
> |1|ben|18|
> |2|ken|19|
> |3|mark|20|
>
> FlinkSQL lookup join with lookup cahce
> {code:java}
> CREATE TABLE dim_user (
> rowkey STRING,
> info ROW,
> PRIMARY KEY (rowkey) NOT ENFORCED
> ) WITH (
> 'connector' = 'hbase-2.2',
> 'zookeeper.quorum' = 'localhost:2181',
> 'zookeeper.znode.parent' = '/hbase',
> 'table-name' = 'default:test',
> 'lookup.cache' = 'PARTIAL',
> 'lookup.partial-cache.max-rows' = '1000',
> 'lookup.partial-cache.expire-after-write' = '1h'
> );
> CREATE VIEW user_click AS
> SELECT user_id, proctime() AS proc_time
> FROM (
> VALUES('1'), ('2'), ('3'), ('1'), ('2')
> ) AS t (user_id);
> SELECT
> user_id,
> info.name,
> info.age
> FROM user_click INNER JOIN dim_user
> FOR SYSTEM_TIME AS OF user_click.proc_time
> ON dim_user.rowkey = user_click.user_id;{code}
>
> Expect Result
> ||rowkey||name||age||
> |1|ben|18|
> |2|ken|19|
> |3|mark|20|
> |1|ben|18|
> |2|ken|19|
>
> Actual Result
> ||rowkey||name||age||
> |1|ben|18|
> |2|ken|19|
> |3|mark|20|
> |1|mark|20|
> |2|mark|20|
>
> Wrong result when we lookup user_id 1 and 2 the second time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-35233] Fix lookup cache reuse RowData object problem [flink-connector-hbase]
Tan-JiaLiang opened a new pull request, #47: URL: https://github.com/apache/flink-connector-hbase/pull/47 ## What is the purpose of the change HBase lookup result is wrong when lookup cache is enabled. ## Brief change log - *fix lookup cache reuse RowData object problem* ## Verifying this change This change is already covered by existing tests, such as - *org.apache.flink.connector.hbase2.HBaseConnectorITCase#testHBaseLookupTableSource* - *org.apache.flink.connector.hbase2.HBaseConnectorITCase#testHBaseAsyncLookupTableSource* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35047][state] Support ForStStateBackend and ForStKeyedStateBackend [flink]
masteryhx commented on code in PR #24682:
URL: https://github.com/apache/flink/pull/24682#discussion_r1578773304
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStKeyedStateBackend.java:
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.util.Disposable;
+import org.apache.flink.util.IOUtils;
+import org.apache.flink.util.Preconditions;
+
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.RocksDB;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.net.URI;
+
+/**
+ * A KeyedStateBackend that stores its state in {@code ForSt}. This state
backend can store very
+ * large state that exceeds memory even disk to remote storage. TODO: Support
to implement the new
+ * interface of KeyedStateBackend
+ */
+public class ForStKeyedStateBackend implements Disposable {
Review Comment:
It will be resolved in
[FLINK-35048](https://issues.apache.org/jira/browse/FLINK-35048) as TODO
described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35047][state] Support ForStStateBackend and ForStKeyedStateBackend [flink]
masteryhx commented on code in PR #24682:
URL: https://github.com/apache/flink/pull/24682#discussion_r1578772481
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStResourceContainer.java:
##
@@ -0,0 +1,547 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.runtime.memory.OpaqueMemoryResource;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.IOUtils;
+import org.apache.flink.util.Preconditions;
+
+import org.rocksdb.BlockBasedTableConfig;
+import org.rocksdb.BloomFilter;
+import org.rocksdb.Cache;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Filter;
+import org.rocksdb.FlinkEnv;
+import org.rocksdb.IndexType;
+import org.rocksdb.PlainTableConfig;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.Statistics;
+import org.rocksdb.TableFormatConfig;
+import org.rocksdb.WriteOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+
+/**
+ * The container for ForSt resources, including option factory and shared
resource among instances.
+ *
+ * This should be the only entrance for ForStStateBackend to get ForSt
options, and should be
+ * properly (and necessarily) closed to prevent resource leak.
+ */
+public final class ForStResourceContainer implements AutoCloseable {
+private static final Logger LOG =
LoggerFactory.getLogger(ForStResourceContainer.class);
+
+private static final String FORST_RELOCATE_LOG_SUFFIX = "_LOG";
+
+// the filename length limit is 255 on most operating systems
+private static final int INSTANCE_PATH_LENGTH_LIMIT = 255 -
FORST_RELOCATE_LOG_SUFFIX.length();
+
+private static final String DB_DIR_STRING = "db";
+
+@Nullable private final URI remoteBasePath;
+
+@Nullable private final URI remoteForStPath;
+
+@Nullable private final File localBasePath;
Review Comment:
`localBasePath` is the path of keyed state backend while `localForStPath` is
the path of ForStDB.
It's also consistent with the design of RocksDB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]
tchivs commented on PR #23629: URL: https://github.com/apache/flink/pull/23629#issuecomment-2076205254 Sure enough it can be used -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35047][state] Support ForStStateBackend and ForStKeyedStateBackend [flink]
masteryhx commented on code in PR #24682:
URL: https://github.com/apache/flink/pull/24682#discussion_r1578770992
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/restore/ForStRestoreOperation.java:
##
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst.restore;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.RestoreOperation;
+
+/** Interface for ForSt restore. */
+@Internal
+public interface ForStRestoreOperation extends
RestoreOperation, AutoCloseable {
Review Comment:
It's just introduced to init the ForStKeyedStateBackend.
I'd prefer to maintain it as `NonRestoreOperation` which is consistent with
other state backends.
We could expand it when we implement real restore operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
Zakelly commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1578769079
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -220,12 +283,17 @@ void insertBlockingBuffer(StateRequest request) {
* @param force whether to trigger requests in force.
*/
void triggerIfNeeded(boolean force) {
-// TODO: introduce a timeout mechanism for triggering.
if (!force && stateRequestsBuffer.activeQueueSize() < batchSize) {
+// if the active queue size is 1, it means that the current
request is the oldest one in
+// the active queue.
+if (stateRequestsBuffer.activeQueueSize() == 1) {
+scheduleTimeout(currentTriggerSeq.get());
+}
Review Comment:
Ah... I've mispoken... Is it possible `scheduleTimeout` multiple times for
one seq?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-35233) HBase lookup result is wrong when lookup cache is enabled
tanjialiang created FLINK-35233:
---
Summary: HBase lookup result is wrong when lookup cache is enabled
Key: FLINK-35233
URL: https://issues.apache.org/jira/browse/FLINK-35233
Project: Flink
Issue Type: Bug
Components: Connectors / HBase
Affects Versions: hbase-3.0.0
Reporter: tanjialiang
HBase table
||rowkey||name||age||
|1|ben|18|
|2|ken|19|
|3|mark|20|
FlinkSQL lookup join with lookup cahce
{code:java}
CREATE TABLE dim_user (
rowkey STRING,
info ROW,
PRIMARY KEY (rowkey) NOT ENFORCED
) WITH (
'connector' = 'hbase-2.2',
'zookeeper.quorum' = 'localhost:2181',
'zookeeper.znode.parent' = '/hbase',
'table-name' = 'default:test',
'lookup.cache' = 'PARTIAL',
'lookup.partial-cache.max-rows' = '1000',
'lookup.partial-cache.expire-after-write' = '1h'
);
CREATE VIEW user_click AS
SELECT user_id, proctime() AS proc_time
FROM (
VALUES('1'), ('2'), ('3'), ('1'), ('2')
) AS t (user_id);
SELECT
user_id,
info.name,
info.age
FROM user_click INNER JOIN dim_user
FOR SYSTEM_TIME AS OF user_click.proc_time
ON dim_user.rowkey = user_click.user_id;{code}
Expect Result
||rowkey||name||age||
|1|ben|18|
|2|ken|19|
|3|mark|20|
|1|ben|18|
|2|ken|19|
Actual Result
||rowkey||name||age||
|1|ben|18|
|2|ken|19|
|3|mark|20|
|1|mark|20|
|2|mark|20|
Wrong result when we lookup user_id 1 and 2 the second time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35208][python] Respect pipeline.cached-files during handling Python dependencies [flink]
flinkbot commented on PR #24716: URL: https://github.com/apache/flink/pull/24716#issuecomment-2076199028 ## CI report: * 127a521041da7a6d13a031ef7e17994c71f30571 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
Zakelly commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1578767927
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -94,29 +106,80 @@ public class AsyncExecutionController {
*/
final AtomicInteger inFlightRecordNum;
-public AsyncExecutionController(MailboxExecutor mailboxExecutor,
StateExecutor stateExecutor) {
-this(mailboxExecutor, stateExecutor, DEFAULT_BATCH_SIZE,
DEFAULT_MAX_IN_FLIGHT_RECORD_NUM);
-}
+/** The executor service that schedules and calls the triggers of this
task. */
+ScheduledExecutorService scheduledExecutor;
+
+ScheduledFuture currentScheduledFuture;
+
+/**
+ * The current trigger sequence number, used to distinguish different
triggers. Every time a
+ * trigger occurs, {@code currentTriggerSeq} increases by one.
+ */
+AtomicLong currentTriggerSeq;
public AsyncExecutionController(
MailboxExecutor mailboxExecutor,
StateExecutor stateExecutor,
int batchSize,
+long bufferTimeout,
int maxInFlightRecords) {
this.keyAccountingUnit = new KeyAccountingUnit<>(maxInFlightRecords);
this.mailboxExecutor = mailboxExecutor;
this.stateFutureFactory = new StateFutureFactory<>(this,
mailboxExecutor);
this.stateExecutor = stateExecutor;
this.batchSize = batchSize;
+this.bufferTimeout = bufferTimeout;
this.maxInFlightRecordNum = maxInFlightRecords;
this.stateRequestsBuffer = new StateRequestBuffer<>();
this.inFlightRecordNum = new AtomicInteger(0);
+this.currentTriggerSeq = new AtomicLong(0);
+
+// - initialize buffer timeout ---
+this.currentScheduledFuture = null;
+if (bufferTimeout > 0) {
+this.scheduledExecutor =
+new ScheduledThreadPoolExecutor(
+1, new
ExecutorThreadFactory("AEC-timeout-scheduler"));
+((ScheduledThreadPoolExecutor)
this.scheduledExecutor).setRemoveOnCancelPolicy(true);
+// make sure shutdown removes all pending tasks
+((ScheduledThreadPoolExecutor) this.scheduledExecutor)
+
.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
+((ScheduledThreadPoolExecutor) this.scheduledExecutor)
+.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
+} else {
+this.scheduledExecutor = null;
+}
+
LOG.info(
-"Create AsyncExecutionController: batchSize {},
maxInFlightRecordsNum {}",
+"Create AsyncExecutionController: batchSize {}, bufferTimeout
{}, maxInFlightRecordsNum {}",
batchSize,
+bufferTimeout,
maxInFlightRecords);
}
+void scheduleTimeout(long triggerSeq) {
+if (bufferTimeout > 0) {
+if (currentScheduledFuture != null
+&& !currentScheduledFuture.isDone()
+&& !currentScheduledFuture.isCancelled()) {
+currentScheduledFuture.cancel(false);
+}
+currentScheduledFuture =
+(ScheduledFuture)
+scheduledExecutor.schedule(
+() -> {
+if (triggerSeq !=
currentTriggerSeq.get()) {
+// if any new trigger occurs, skip
this schedule
+return;
+}
+mailboxExecutor.execute(
+() -> triggerIfNeeded(true),
"AEC-timeout");
Review Comment:
Well, I mean we keep the `if (triggerSeq != currentTriggerSeq.get()) {
return; }` before this, only change the mailbox processing part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35162][state] Implement WriteBatchOperation and general MultiGetOperation for ForSt [flink]
ljz2051 commented on code in PR #24681:
URL: https://github.com/apache/flink/pull/24681#discussion_r1578764009
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStWriteBatchOperation.java:
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst;
+
+import org.rocksdb.RocksDB;
+import org.rocksdb.WriteBatch;
+import org.rocksdb.WriteOptions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * The writeBatch operation implementation for ForStDB.
+ *
+ * @param The type of key in put access request.
+ * @param The type of value in put access request.
+ */
+public class ForStWriteBatchOperation implements ForStDBOperation {
+
+private static final int PER_RECORD_ESTIMATE_BYTES = 100;
+
+private final RocksDB db;
+
+private final List> batchRequest;
+
+private final WriteOptions writeOptions;
+
+ForStWriteBatchOperation(
+RocksDB db, List> batchRequest, WriteOptions
writeOptions) {
+this.db = db;
+this.batchRequest = batchRequest;
+this.writeOptions = writeOptions;
+}
+
+@Override
+public CompletableFuture process() throws IOException {
Review Comment:
I have refined it !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [FLINK-35208][python] Respect pipeline.cached-files during handling Python dependencies [flink]
dianfu opened a new pull request, #24716: URL: https://github.com/apache/flink/pull/24716 … ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing). *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35208) Respect pipeline.cached-files during processing Python dependencies
[ https://issues.apache.org/jira/browse/FLINK-35208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35208: --- Labels: pull-request-available (was: ) > Respect pipeline.cached-files during processing Python dependencies > --- > > Key: FLINK-35208 > URL: https://issues.apache.org/jira/browse/FLINK-35208 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > > Currently, PyFlink will make use of distributed cache > (StreamExecutionEnvironment#cachedFiles) during handling the Python > dependencies(See > [https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/python/util/PythonDependencyUtils.java#L339] > for more details). > However, if pipeline.cached-files is configured, it will clear > StreamExecutionEnvironment#cachedFiles(see > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L1132] > for more details) which may break the above functionalities. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-29050][JUnit5 Migration] Module: flink-hadoop-compatibility [flink]
RocMarshal commented on PR #20990: URL: https://github.com/apache/flink/pull/20990#issuecomment-2076180251 Thanks all of you related about the PR. 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
[ https://issues.apache.org/jira/browse/FLINK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-33463: --- Assignee: RocMarshal > Support the implementation of dynamic source tables based on the new source > --- > > Key: FLINK-33463 > URL: https://issues.apache.org/jira/browse/FLINK-33463 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33462) Sort out the document page about the new Jdbc source.
[ https://issues.apache.org/jira/browse/FLINK-33462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-33462: --- Assignee: RocMarshal > Sort out the document page about the new Jdbc source. > - > > Key: FLINK-33462 > URL: https://issues.apache.org/jira/browse/FLINK-33462 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33461) Support streaming related semantics for the new jdbc source
[ https://issues.apache.org/jira/browse/FLINK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-33461: --- Assignee: RocMarshal > Support streaming related semantics for the new jdbc source > --- > > Key: FLINK-33461 > URL: https://issues.apache.org/jira/browse/FLINK-33461 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35230) Split FlinkSqlParserImplTest to reduce the code lines.
[
https://issues.apache.org/jira/browse/FLINK-35230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840623#comment-17840623
]
Shuai Xu commented on FLINK-35230:
--
[~lsy] I'd like to take this, would you assign it to me?
> Split FlinkSqlParserImplTest to reduce the code lines.
> --
>
> Key: FLINK-35230
> URL: https://issues.apache.org/jira/browse/FLINK-35230
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / API
>Reporter: Feng Jin
>Priority: Major
>
> With the increasing extension of Calcite syntax, the current
> FlinkSqlParserImplTest has reached nearly 3000 lines of code.
> If it exceeds the current limit, it will result in errors in the code style
> check.
> {code:java}
> Unable to find source-code formatter for language: log. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yaml08:33:19.679 [ERROR]
> src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java:[1]
> (sizes) FileLength: File length is 3,166 lines (max allowed is 3,100).
> {code}
> To facilitate future syntax extends, I suggest that we split
> FlinkSqlParserImplTest and place the same type of syntax in separate Java
> tests for the convenience of avoiding the continuous growth of the original
> test class.
> My current idea is:
> Since *FlinkSqlParserImplTest* currently inherits {*}SqlParserTest{*}, and
> *SqlParserTest* itself contains many unit tests, for the convenience of
> future test splits, we should introduce a basic *ParserTestBase* inheriting
> {*}SqlParserTest{*}, and disable the original related unit tests in
> {*}SqlParserTest{*}.
> This will facilitate writing relevant unit tests more quickly during
> subsequent splitting, without the need to repeatedly execute the unit tests
> inside SqlParserTest.
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59113&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35176) Support property authentication connection for JDBC catalog & dynamic table
[ https://issues.apache.org/jira/browse/FLINK-35176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-35176: --- Assignee: RocMarshal > Support property authentication connection for JDBC catalog & dynamic table > --- > > Key: FLINK-35176 > URL: https://issues.apache.org/jira/browse/FLINK-35176 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35230) Split FlinkSqlParserImplTest to reduce the code lines.
[
https://issues.apache.org/jira/browse/FLINK-35230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dalongliu updated FLINK-35230:
--
Component/s: Table SQL / API
(was: Table SQL / Planner)
> Split FlinkSqlParserImplTest to reduce the code lines.
> --
>
> Key: FLINK-35230
> URL: https://issues.apache.org/jira/browse/FLINK-35230
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / API
>Reporter: Feng Jin
>Priority: Major
>
> With the increasing extension of Calcite syntax, the current
> FlinkSqlParserImplTest has reached nearly 3000 lines of code.
> If it exceeds the current limit, it will result in errors in the code style
> check.
> {code:java}
> Unable to find source-code formatter for language: log. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yaml08:33:19.679 [ERROR]
> src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java:[1]
> (sizes) FileLength: File length is 3,166 lines (max allowed is 3,100).
> {code}
> To facilitate future syntax extends, I suggest that we split
> FlinkSqlParserImplTest and place the same type of syntax in separate Java
> tests for the convenience of avoiding the continuous growth of the original
> test class.
> My current idea is:
> Since *FlinkSqlParserImplTest* currently inherits {*}SqlParserTest{*}, and
> *SqlParserTest* itself contains many unit tests, for the convenience of
> future test splits, we should introduce a basic *ParserTestBase* inheriting
> {*}SqlParserTest{*}, and disable the original related unit tests in
> {*}SqlParserTest{*}.
> This will facilitate writing relevant unit tests more quickly during
> subsequent splitting, without the need to repeatedly execute the unit tests
> inside SqlParserTest.
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59113&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-33460) Support property authentication connection.
[ https://issues.apache.org/jira/browse/FLINK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan resolved FLINK-33460. - Resolution: Fixed > Support property authentication connection. > --- > > Key: FLINK-33460 > URL: https://issues.apache.org/jira/browse/FLINK-33460 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > Fix For: jdbc-3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35176) Support property authentication connection for JDBC catalog & dynamic table
[ https://issues.apache.org/jira/browse/FLINK-35176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35176: Fix Version/s: jdbc-3.3.0 > Support property authentication connection for JDBC catalog & dynamic table > --- > > Key: FLINK-35176 > URL: https://issues.apache.org/jira/browse/FLINK-35176 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > Labels: pull-request-available > Fix For: jdbc-3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35230) Split FlinkSqlParserImplTest to reduce the code lines.
[
https://issues.apache.org/jira/browse/FLINK-35230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840620#comment-17840620
]
dalongliu commented on FLINK-35230:
---
Good point, +1.
> Split FlinkSqlParserImplTest to reduce the code lines.
> --
>
> Key: FLINK-35230
> URL: https://issues.apache.org/jira/browse/FLINK-35230
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / Planner
>Reporter: Feng Jin
>Priority: Major
>
> With the increasing extension of Calcite syntax, the current
> FlinkSqlParserImplTest has reached nearly 3000 lines of code.
> If it exceeds the current limit, it will result in errors in the code style
> check.
> {code:java}
> Unable to find source-code formatter for language: log. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yaml08:33:19.679 [ERROR]
> src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java:[1]
> (sizes) FileLength: File length is 3,166 lines (max allowed is 3,100).
> {code}
> To facilitate future syntax extends, I suggest that we split
> FlinkSqlParserImplTest and place the same type of syntax in separate Java
> tests for the convenience of avoiding the continuous growth of the original
> test class.
> My current idea is:
> Since *FlinkSqlParserImplTest* currently inherits {*}SqlParserTest{*}, and
> *SqlParserTest* itself contains many unit tests, for the convenience of
> future test splits, we should introduce a basic *ParserTestBase* inheriting
> {*}SqlParserTest{*}, and disable the original related unit tests in
> {*}SqlParserTest{*}.
> This will facilitate writing relevant unit tests more quickly during
> subsequent splitting, without the need to repeatedly execute the unit tests
> inside SqlParserTest.
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59113&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan resolved FLINK-29050. - Fix Version/s: 1.20.0 Resolution: Fixed > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available, stale-assigned, starter > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33460) Support property authentication connection.
[ https://issues.apache.org/jira/browse/FLINK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-33460: Fix Version/s: jdbc-3.3.0 > Support property authentication connection. > --- > > Key: FLINK-33460 > URL: https://issues.apache.org/jira/browse/FLINK-33460 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > Fix For: jdbc-3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840618#comment-17840618 ] Rui Fan commented on FLINK-29050: - Merged to master(1.20.0) via: * ffa639a31c769880dceb39234700a75f0945ec65 * 29a045574efd498ba0cb760845a459928c722531 * 92eef24d4cc531d6474252ef909fc6d431285dd9 > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available, stale-assigned, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-29050][JUnit5 Migration] Module: flink-hadoop-compatibility [flink]
1996fanrui merged PR #20990: URL: https://github.com/apache/flink/pull/20990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33460) Support property authentication connection.
[ https://issues.apache.org/jira/browse/FLINK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840615#comment-17840615 ] Rui Fan commented on FLINK-33460: - Merged to main via bb43b3c18fda6a2e587b3a3941d697803fe86441 > Support property authentication connection. > --- > > Key: FLINK-33460 > URL: https://issues.apache.org/jira/browse/FLINK-33460 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33460) Support property authentication connection.
[ https://issues.apache.org/jira/browse/FLINK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-33460: --- Assignee: RocMarshal > Support property authentication connection. > --- > > Key: FLINK-33460 > URL: https://issues.apache.org/jira/browse/FLINK-33460 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33460][Connector/JDBC] Support property authentication connection. [flink-connector-jdbc]
1996fanrui merged PR #115: URL: https://github.com/apache/flink-connector-jdbc/pull/115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35189) Introduce test-filesystem Catalog based on FileSystem Connector to support materialized table
[ https://issues.apache.org/jira/browse/FLINK-35189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu updated FLINK-35189: -- Component/s: Tests > Introduce test-filesystem Catalog based on FileSystem Connector to support > materialized table > - > > Key: FLINK-35189 > URL: https://issues.apache.org/jira/browse/FLINK-35189 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Tests >Affects Versions: 1.20.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35188) Introduce CatalogMaterializedTable and related interface to support materialized table
[ https://issues.apache.org/jira/browse/FLINK-35188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840613#comment-17840613 ] dalongliu commented on FLINK-35188: --- Merged in master: 8576178c4c084c38897e395479df11f15a4ea402 > Introduce CatalogMaterializedTable and related interface to support > materialized table > -- > > Key: FLINK-35188 > URL: https://issues.apache.org/jira/browse/FLINK-35188 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.20.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-35188) Introduce CatalogMaterializedTable and related interface to support materialized table
[ https://issues.apache.org/jira/browse/FLINK-35188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu resolved FLINK-35188. --- Resolution: Fixed > Introduce CatalogMaterializedTable and related interface to support > materialized table > -- > > Key: FLINK-35188 > URL: https://issues.apache.org/jira/browse/FLINK-35188 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.20.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-35088) watermark alignment maxAllowedWatermarkDrift and updateInterval param need check
[
https://issues.apache.org/jira/browse/FLINK-35088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan resolved FLINK-35088.
-
Fix Version/s: 1.20.0
Resolution: Fixed
> watermark alignment maxAllowedWatermarkDrift and updateInterval param need
> check
>
>
> Key: FLINK-35088
> URL: https://issues.apache.org/jira/browse/FLINK-35088
> Project: Flink
> Issue Type: Improvement
> Components: API / Core, Runtime / Coordination
>Affects Versions: 1.16.1
>Reporter: elon_X
>Assignee: elon_X
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Attachments: image-2024-04-11-20-12-29-951.png
>
>
> When I use watermark alignment,
> 1.I found that setting maxAllowedWatermarkDrift to a negative number
> initially led me to believe it could support delaying the consumption of the
> source, so I tried it. Then, the upstream data flow would hang indefinitely.
> Root cause:
> {code:java}
> long maxAllowedWatermark = globalCombinedWatermark.getTimestamp()
> + watermarkAlignmentParams.getMaxAllowedWatermarkDrift(); {code}
> If maxAllowedWatermarkDrift is negative, SourceOperator: maxAllowedWatermark
> < lastEmittedWatermark, then the SourceReader will be blocked indefinitely
> and cannot recover.
> I'm not sure if this is a supported feature of watermark alignment. If it's
> not, I think an additional parameter validation should be implemented to
> throw an exception on the client side if the value is negative.
> 2.The updateInterval parameter also lacks validation. If I set it to 0, the
> task will throw an exception when starting the job manager. The JDK class
> java.util.concurrent.ScheduledThreadPoolExecutor performs the validation and
> throws the exception.
> {code:java}
> java.lang.IllegalArgumentException: null
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:565)
> ~[?:1.8.0_351]
> at
> org.apache.flink.runtime.source.coordinator.SourceCoordinator.(SourceCoordinator.java:191)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.source.coordinator.SourceCoordinatorProvider.getCoordinator(SourceCoordinatorProvider.java:92)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.createNewInternalCoordinator(RecreateOnResetOperatorCoordinator.java:333)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.(RecreateOnResetOperatorCoordinator.java:59)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.(RecreateOnResetOperatorCoordinator.java:42)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$Provider.create(RecreateOnResetOperatorCoordinator.java:201)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$Provider.create(RecreateOnResetOperatorCoordinator.java:195)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.create(OperatorCoordinatorHolder.java:529)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.create(OperatorCoordinatorHolder.java:494)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.createOperatorCoordinatorHolder(ExecutionJobVertex.java:286)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.initialize(ExecutionJobVertex.java:223)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.initializeJobVertex(DefaultExecutionGraph.java:901)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.initializeJobVertices(DefaultExecutionGraph.java:891)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:848)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:830)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:203)
> ~[flink-dist_2
[jira] [Commented] (FLINK-35088) watermark alignment maxAllowedWatermarkDrift and updateInterval param need check
[
https://issues.apache.org/jira/browse/FLINK-35088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840610#comment-17840610
]
Rui Fan commented on FLINK-35088:
-
Merged to master(1.20) via: 09531994e3cde35ec159315389cf20396c8e21bd
> watermark alignment maxAllowedWatermarkDrift and updateInterval param need
> check
>
>
> Key: FLINK-35088
> URL: https://issues.apache.org/jira/browse/FLINK-35088
> Project: Flink
> Issue Type: Improvement
> Components: API / Core, Runtime / Coordination
>Affects Versions: 1.16.1
>Reporter: elon_X
>Assignee: elon_X
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-04-11-20-12-29-951.png
>
>
> When I use watermark alignment,
> 1.I found that setting maxAllowedWatermarkDrift to a negative number
> initially led me to believe it could support delaying the consumption of the
> source, so I tried it. Then, the upstream data flow would hang indefinitely.
> Root cause:
> {code:java}
> long maxAllowedWatermark = globalCombinedWatermark.getTimestamp()
> + watermarkAlignmentParams.getMaxAllowedWatermarkDrift(); {code}
> If maxAllowedWatermarkDrift is negative, SourceOperator: maxAllowedWatermark
> < lastEmittedWatermark, then the SourceReader will be blocked indefinitely
> and cannot recover.
> I'm not sure if this is a supported feature of watermark alignment. If it's
> not, I think an additional parameter validation should be implemented to
> throw an exception on the client side if the value is negative.
> 2.The updateInterval parameter also lacks validation. If I set it to 0, the
> task will throw an exception when starting the job manager. The JDK class
> java.util.concurrent.ScheduledThreadPoolExecutor performs the validation and
> throws the exception.
> {code:java}
> java.lang.IllegalArgumentException: null
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:565)
> ~[?:1.8.0_351]
> at
> org.apache.flink.runtime.source.coordinator.SourceCoordinator.(SourceCoordinator.java:191)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.source.coordinator.SourceCoordinatorProvider.getCoordinator(SourceCoordinatorProvider.java:92)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.createNewInternalCoordinator(RecreateOnResetOperatorCoordinator.java:333)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.(RecreateOnResetOperatorCoordinator.java:59)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.(RecreateOnResetOperatorCoordinator.java:42)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$Provider.create(RecreateOnResetOperatorCoordinator.java:201)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$Provider.create(RecreateOnResetOperatorCoordinator.java:195)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.create(OperatorCoordinatorHolder.java:529)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.create(OperatorCoordinatorHolder.java:494)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.createOperatorCoordinatorHolder(ExecutionJobVertex.java:286)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.initialize(ExecutionJobVertex.java:223)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.initializeJobVertex(DefaultExecutionGraph.java:901)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.initializeJobVertices(DefaultExecutionGraph.java:891)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:848)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:830)
> ~[flink-dist_2.12-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecuti
Re: [PR] [FLINK-35088][core] Watermark alignment maxAllowedWatermarkDrift and … [flink]
1996fanrui merged PR #24706: URL: https://github.com/apache/flink/pull/24706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35088][core] Watermark alignment maxAllowedWatermarkDrift and … [flink]
1996fanrui commented on PR #24706: URL: https://github.com/apache/flink/pull/24706#issuecomment-2076132786 CI is green for now, merging~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] Use jdk 1.8 for profile activation [flink-connector-opensearch]
snuyanzin merged PR #45: URL: https://github.com/apache/flink-connector-opensearch/pull/45 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump org.opensearch:opensearch from 1.3.0 to 2.11.1 in /flink-connector-opensearch [flink-connector-opensearch]
dependabot[bot] commented on PR #37: URL: https://github.com/apache/flink-connector-opensearch/pull/37#issuecomment-2075963822 OK, I won't notify you about org.opensearch:opensearch again, unless you re-open this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump org.opensearch:opensearch from 1.3.0 to 2.11.1 in /flink-connector-opensearch [flink-connector-opensearch]
snuyanzin commented on PR #37: URL: https://github.com/apache/flink-connector-opensearch/pull/37#issuecomment-2075963770 @dependabot ignore this dependency we need both -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump org.opensearch:opensearch from 1.3.0 to 2.11.1 in /flink-connector-opensearch [flink-connector-opensearch]
dependabot[bot] closed pull request #37: Bump org.opensearch:opensearch from 1.3.0 to 2.11.1 in /flink-connector-opensearch URL: https://github.com/apache/flink-connector-opensearch/pull/37 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35228) DynamicKafkaSource does not read re-added topic for the same cluster
[ https://issues.apache.org/jira/browse/FLINK-35228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mason Chen reassigned FLINK-35228: -- Assignee: Ignas Daukšas > DynamicKafkaSource does not read re-added topic for the same cluster > > > Key: FLINK-35228 > URL: https://issues.apache.org/jira/browse/FLINK-35228 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: kafka-3.1.0 >Reporter: Ignas Daukšas >Assignee: Ignas Daukšas >Priority: Major > Labels: pull-request-available > > When using DynamicKafkaSource, if topic is removed from the cluster (that has > more active topics remaining) and then re-added back, consumption from that > topic won't be happening. > However, if the topic in question is the only topic in that cluster, then > everything works as expected - consumption restarts once cluster-topic is > re-added. > Steps to reproduce: > # Have a DynamicKafkaSource. > # Have KafkaMetadataService report a single cluster with two topics (A and > B) for the subscribed stream/streams. > # Consume some data, topics A and B are consumed as expected. > # Have KafkaMetadataService remove topic A. > # Continue consuming data, only topic B consumed as expected. > # Have KafkaMetadataService re-add topic A. > # Continue consuming data, however only topic B is actually consumed - this > is not expected. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33859] Support OpenSearch v2 [flink-connector-opensearch]
snuyanzin merged PR #38: URL: https://github.com/apache/flink-connector-opensearch/pull/38 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33859] Support OpenSearch v2 [flink-connector-opensearch]
snuyanzin commented on PR #38: URL: https://github.com/apache/flink-connector-opensearch/pull/38#issuecomment-2075783174 Thanks for the review I solved the conflicts and added profiles to simplify releasing separately for v1 and v2 as was mentioned in ml -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas M updated FLINK-35232: Labels: (was: pull-request-available) > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. While setting the params enabled > here reduced read timeouts, we still see 503 errors leading to Flink job > restarts. > Thus, in this ticket, we want to specify additional retry settings as noted > in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We want > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas M updated FLINK-35232: Affects Version/s: 1.18.1 1.19.0 > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > Labels: pull-request-available > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. While setting the params enabled > here reduced read timeouts, we still see 503 errors leading to Flink job > restarts. > Thus, in this ticket, we want to specify additional retry settings as noted > in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We want > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas M updated FLINK-35232: Description: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts. Thus, in this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We want [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. was: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. In this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We want [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. The current GCS connector uses the gcs java storage library and bypasses the hadoop gcs connector which supports multiple http options. There are situations where GCS takes longer to provide a response for a PUT operation than the default value. This change will allow users to customize their connect time and read timeout based on their application > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > Labels: pull-request-available > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. While setting the params enabled > here reduced read timeouts, we still see 503 errors leading to Flink job > restarts. > Thus, in this ticket, we want to specify additional retry settings as noted > in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We want > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas M updated FLINK-35232: Description: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. In this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We want [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. The current GCS connector uses the gcs java storage library and bypasses the hadoop gcs connector which supports multiple http options. There are situations where GCS takes longer to provide a response for a PUT operation than the default value. This change will allow users to customize their connect time and read timeout based on their application was: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. In this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries] The current GCS connector uses the gcs java storage library and bypasses the hadoop gcs connector which supports multiple http options. There are situations where GCS takes longer to provide a response for a PUT operation than the default value. This change will allow users to customize their connect time and read timeout based on their application > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > Labels: pull-request-available > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. > In this ticket, we want to specify additional retry settings as noted in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We want > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. > The current GCS connector uses the gcs java storage library and bypasses the > hadoop gcs connector which supports multiple http options. There are > situations where GCS takes longer to provide a response for a PUT operation > than the default value. > This change will allow users to customize their connect time and read timeout > based on their application -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas M updated FLINK-35232: Description: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. In this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries] The current GCS connector uses the gcs java storage library and bypasses the hadoop gcs connector which supports multiple http options. There are situations where GCS takes longer to provide a response for a PUT operation than the default value. This change will allow users to customize their connect time and read timeout based on their application was: The current GCS connector uses the gcs java storage library and bypasses the hadoop gcs connector which supports multiple http options. There are situations where GCS takes longer to provide a response for a PUT operation than the default value. This change will allow users to customize their connect time and read timeout based on their application > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > Labels: pull-request-available > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. > In this ticket, we want to specify additional retry settings as noted in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > The current GCS connector uses the gcs java storage library and bypasses the > hadoop gcs connector which supports multiple http options. There are > situations where GCS takes longer to provide a response for a PUT operation > than the default value. > This change will allow users to customize their connect time and read timeout > based on their application -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35232) Support for retry settings on GCS connector
Vikas M created FLINK-35232: --- Summary: Support for retry settings on GCS connector Key: FLINK-35232 URL: https://issues.apache.org/jira/browse/FLINK-35232 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem Affects Versions: 1.15.3, 1.16.2, 1.17.1 Reporter: Vikas M Assignee: Ravi Singh The current GCS connector uses the gcs java storage library and bypasses the hadoop gcs connector which supports multiple http options. There are situations where GCS takes longer to provide a response for a PUT operation than the default value. This change will allow users to customize their connect time and read timeout based on their application -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35231) Tests rely on listJobsFunction in TestingClusterClient are broken
[ https://issues.apache.org/jira/browse/FLINK-35231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Plyusnin updated FLINK-35231: --- Summary: Tests rely on listJobsFunction in TestingClusterClient are broken (was: Test relies on listJobsFunction in TestingClusterClient are broken) > Tests rely on listJobsFunction in TestingClusterClient are broken > - > > Key: FLINK-35231 > URL: https://issues.apache.org/jira/browse/FLINK-35231 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.9.0 >Reporter: Artem Plyusnin >Priority: Major > > Supplier listJobsFunction is never called in TestingClusterClient therefore > some assertions are never called. > Affected tests: > * FlinkSessionJobObserverTest#testObserveWithEffectiveConfig > * FlinkDeploymentControllerTest#verifyReconcileWithBadConfig > * FlinkSessionJobControllerTest#verifyReconcileWithBadConfig > Reason: > When listJobs() requested TestingClusterClient calls > TestingFlinkClient#getMultipleJobsDetails instead of listJobsFunction -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35231) Test relies on listJobsFunction in TestingClusterClient are broken
Artem Plyusnin created FLINK-35231: -- Summary: Test relies on listJobsFunction in TestingClusterClient are broken Key: FLINK-35231 URL: https://issues.apache.org/jira/browse/FLINK-35231 Project: Flink Issue Type: Bug Components: Kubernetes Operator Affects Versions: kubernetes-operator-1.9.0 Reporter: Artem Plyusnin Supplier listJobsFunction is never called in TestingClusterClient therefore some assertions are never called. Affected tests: * FlinkSessionJobObserverTest#testObserveWithEffectiveConfig * FlinkDeploymentControllerTest#verifyReconcileWithBadConfig * FlinkSessionJobControllerTest#verifyReconcileWithBadConfig Reason: When listJobs() requested TestingClusterClient calls TestingFlinkClient#getMultipleJobsDetails instead of listJobsFunction -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Draft pr for initial review of apicurio [flink]
flinkbot commented on PR #24715: URL: https://github.com/apache/flink/pull/24715#issuecomment-2075321251 ## CI report: * 0fdfcddfaed3a5fb73e6bbb01059073f90e5e587 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [cdc-connector][db2] Db2 support incremental source [flink-cdc]
leonardBang merged PR #2870: URL: https://github.com/apache/flink-cdc/pull/2870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [cdc-connector][db2] Db2 support incremental source [flink-cdc]
leonardBang commented on PR #2870: URL: https://github.com/apache/flink-cdc/pull/2870#issuecomment-2075317910 CI passed, merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
