[PR] [docs] Delete redundant note [flink-cdc]
Gianzie opened a new pull request, #3384: URL: https://github.com/apache/flink-cdc/pull/3384 Delete redundant note -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34545][cdc-pipeline-connector]Add OceanBase pipeline connector to Flink CDC [flink-cdc]
yuanoOo commented on PR #3360: URL: https://github.com/apache/flink-cdc/pull/3360#issuecomment-2141342325 Hello everyone, Is there anyone currently following this PR and willing to do a code review? I will continue to follow up on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] FLINK-35402 [flink-operator][Deployment] add labels to metadata [flink-kubernetes-operator]
gyfora commented on code in PR #829:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/829#discussion_r1621803625
##
helm/flink-kubernetes-operator/templates/flink-operator.yaml:
##
@@ -23,6 +23,11 @@ metadata:
namespace: {{ .Release.Namespace }}
labels:
{{- include "flink-operator.labels" . | nindent 4 }}
Review Comment:
Ok in that case you need to update the respective doc pages too please, as
they would be incorrect:
https://github.com/apache/flink-kubernetes-operator/assets/5880972/88773d2a-1021-4ab6-bf46-c2aee2a30421";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [doc]Flink sql gettingstarted Expression 'dept_id' is not being grouped and sql-queries-overview TableDescriptor Missing required options are: path [flink]
ZmmBigdata commented on PR #24841: URL: https://github.com/apache/flink/pull/24841#issuecomment-2141266594 @reswqa please help take a look at the doc update PR. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source [flink-connector-jdbc]
RocMarshal commented on code in PR #119: URL: https://github.com/apache/flink-connector-jdbc/pull/119#discussion_r1621712488 ## flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/utils/ContinuousUnBoundingSettings.java: ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.connector.jdbc.utils; + +import org.apache.flink.annotation.PublicEvolving; + +import java.io.Serializable; +import java.time.Duration; +import java.util.Objects; + +import static org.apache.flink.util.Preconditions.checkNotNull; + +/** + * Settings describing how to do continuous file discovery and enumeration for the file source's + * continuous discovery and streaming mode. + */ +@PublicEvolving Review Comment: Thanks @1996fanrui for the review. https://github.com/apache/flink-connector-jdbc/assets/64569824/2ad4d5a4-8bf7-4987-8e18-ca079c7a59c1";> I sorted out the rules based on the archunit of flink repo. - For config options: `Experimental` annatation is allowed to use. - For table api: `Experimental` annatation is allowed to use. - For ApiAnnotation: `Experimental` annotation is allowed to use for classes/API that is located in `org.apache.flink..api..` package and not located in `..internal.. ` pacakge I made two draft pr https://github.com/apache/flink-connector-jdbc/pull/125 https://github.com/apache/flink-connector-jdbc/pull/124 in using `Experimental` for checking rule links. It was failed. So, It seems that the `Experimental` annotation is not suitable here, just `PublicEvolving` please let me know what's your opinion. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35406] Use inner serializer when casting RAW type to BINARY or… [flink]
docete commented on code in PR #24818:
URL: https://github.com/apache/flink/pull/24818#discussion_r1621681290
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RawToBinaryCastRule.java:
##
@@ -62,7 +62,7 @@ public boolean canFail(LogicalType inputLogicalType,
LogicalType targetLogicalTy
// new behavior
isNull$290 = isNull$289;
if (!isNull$290) {
-byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292);
+byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292.getInnerSerializer());
Review Comment:
BTW, modify the code generator make things more complicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35406] Use inner serializer when casting RAW type to BINARY or… [flink]
docete commented on code in PR #24818:
URL: https://github.com/apache/flink/pull/24818#discussion_r1621677150
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RawToBinaryCastRule.java:
##
@@ -62,7 +62,7 @@ public boolean canFail(LogicalType inputLogicalType,
LogicalType targetLogicalTy
// new behavior
isNull$290 = isNull$289;
if (!isNull$290) {
-byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292);
+byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292.getInnerSerializer());
Review Comment:
> Why do we actually need the call to `getInnerSerializer()` during runtime
shouldn't we simply use this serializer as the `typeSerializer$292` for code
generation? Or does the implementation not allow this?
I think the RAW type should bind with RawValueDataSerializer, and use
`getInnerSerializer()` in se/de phase. Follow the same pattern also make the
Generated code more clear. See: AbstractBinaryWriter::writeRawValue
```
public void writeRawValue(
int pos, RawValueData input, RawValueDataSerializer
serializer) {
TypeSerializer innerSerializer = serializer.getInnerSerializer();
// RawValueData only has one implementation which is
BinaryRawValueData
BinaryRawValueData rawValue = (BinaryRawValueData) input;
rawValue.ensureMaterialized(innerSerializer);
writeSegmentsToVarLenPart(
pos, rawValue.getSegments(), rawValue.getOffset(),
rawValue.getSizeInBytes());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35406] Use inner serializer when casting RAW type to BINARY or… [flink]
docete commented on code in PR #24818:
URL: https://github.com/apache/flink/pull/24818#discussion_r1621677150
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RawToBinaryCastRule.java:
##
@@ -62,7 +62,7 @@ public boolean canFail(LogicalType inputLogicalType,
LogicalType targetLogicalTy
// new behavior
isNull$290 = isNull$289;
if (!isNull$290) {
-byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292);
+byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292.getInnerSerializer());
Review Comment:
> Why do we actually need the call to `getInnerSerializer()` during runtime
shouldn't we simply use this serializer as the `typeSerializer$292` for code
generation? Or does the implementation not allow this?
I think the RAW type should bind with RawValueDataSerializer, and use
`getInnerSerializer()` in se/de phase. Follow the same pattern also make the
Generated code more clear. See: AbstractBinaryWriter::writeRawValue
`
public void writeRawValue(
int pos, RawValueData input, RawValueDataSerializer
serializer) {
TypeSerializer innerSerializer = serializer.getInnerSerializer();
// RawValueData only has one implementation which is
BinaryRawValueData
BinaryRawValueData rawValue = (BinaryRawValueData) input;
rawValue.ensureMaterialized(innerSerializer);
writeSegmentsToVarLenPart(
pos, rawValue.getSegments(), rawValue.getOffset(),
rawValue.getSizeInBytes());
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35406] Use inner serializer when casting RAW type to BINARY or… [flink]
docete commented on code in PR #24818:
URL: https://github.com/apache/flink/pull/24818#discussion_r1621677150
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RawToBinaryCastRule.java:
##
@@ -62,7 +62,7 @@ public boolean canFail(LogicalType inputLogicalType,
LogicalType targetLogicalTy
// new behavior
isNull$290 = isNull$289;
if (!isNull$290) {
-byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292);
+byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292.getInnerSerializer());
Review Comment:
> Why do we actually need the call to `getInnerSerializer()` during runtime
shouldn't we simply use this serializer as the `typeSerializer$292` for code
generation? Or does the implementation not allow this?
I think the RAW type should bind with RawValueDataSerializer, and use
`getInnerSerializer()` in se/de phase. Follow the same pattern also make the
Generated code more clear. See: `AbstractBinaryWriter::writeRawValue`
`public void writeRawValue(
int pos, RawValueData input, RawValueDataSerializer
serializer) {
TypeSerializer innerSerializer = serializer.getInnerSerializer();
// RawValueData only has one implementation which is
BinaryRawValueData
BinaryRawValueData rawValue = (BinaryRawValueData) input;
rawValue.ensureMaterialized(innerSerializer);
writeSegmentsToVarLenPart(
pos, rawValue.getSegments(), rawValue.getOffset(),
rawValue.getSizeInBytes());
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35406] Use inner serializer when casting RAW type to BINARY or… [flink]
docete commented on code in PR #24818:
URL: https://github.com/apache/flink/pull/24818#discussion_r1621677150
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RawToBinaryCastRule.java:
##
@@ -62,7 +62,7 @@ public boolean canFail(LogicalType inputLogicalType,
LogicalType targetLogicalTy
// new behavior
isNull$290 = isNull$289;
if (!isNull$290) {
-byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292);
+byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292.getInnerSerializer());
Review Comment:
> Why do we actually need the call to `getInnerSerializer()` during runtime
shouldn't we simply use this serializer as the `typeSerializer$292` for code
generation? Or does the implementation not allow this?
I think the RAW type should bind with RawValueDataSerializer, and use
`getInnerSerializer()` in se/de phase. Follow the same pattern also make the
Generated code more clear. See: `AbstractBinaryWriter::writeRawValue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan closed FLINK-35215. --- Resolution: Fixed > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: image-2024-04-25-14-57-55-231.png, > image-2024-04-25-15-00-32-410.png, image-2024-05-31-11-22-49-226.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=200 > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=200 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850904#comment-17850904 ] Rui Fan commented on FLINK-35215: - The performance seems already recovered, let me close this JIRA. !image-2024-05-31-11-22-49-226.png|width=1121,height=389! > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: image-2024-04-25-14-57-55-231.png, > image-2024-04-25-15-00-32-410.png, image-2024-05-31-11-22-49-226.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=200 > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=200 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35215: Description: The performance of serializerKryo and serializerKryoWithoutRegistration are regressed[1][2], I checked recent commits, and found FLINK-34954 changed related logic. [1] http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=200 [2] http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=200 was: The performance of serializerKryo and serializerKryoWithoutRegistration are regressed[1][2], I checked recent commits, and found FLINK-34954 changed related logic. [1] [http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] [2] http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50 > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: image-2024-04-25-14-57-55-231.png, > image-2024-04-25-15-00-32-410.png, image-2024-05-31-11-22-49-226.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=200 > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=200 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35215: Attachment: image-2024-05-31-11-22-49-226.png > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: image-2024-04-25-14-57-55-231.png, > image-2024-04-25-15-00-32-410.png, image-2024-05-31-11-22-49-226.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=200 > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=200 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35295) Improve jdbc connection pool initialization failure message
[ https://issues.apache.org/jira/browse/FLINK-35295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850903#comment-17850903 ] Jiabao Sun commented on FLINK-35295: release-3.1: e18e7a2523ac1ea59471e5714eb60f544e9f4a04 > Improve jdbc connection pool initialization failure message > --- > > Key: FLINK-35295 > URL: https://issues.apache.org/jira/browse/FLINK-35295 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Xiao Huang >Assignee: Xiao Huang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0, cdc-3.1.1 > > > As described in ticket title. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35295) Improve jdbc connection pool initialization failure message
[ https://issues.apache.org/jira/browse/FLINK-35295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun updated FLINK-35295: --- Fix Version/s: cdc-3.1.1 > Improve jdbc connection pool initialization failure message > --- > > Key: FLINK-35295 > URL: https://issues.apache.org/jira/browse/FLINK-35295 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Xiao Huang >Assignee: Xiao Huang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0, cdc-3.1.1 > > > As described in ticket title. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [BP-3.1][FLINK-35295][mysql] Improve jdbc connection pool initialization failure message [flink-cdc]
Jiabao-Sun merged PR #3374: URL: https://github.com/apache/flink-cdc/pull/3374 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35495) The native metrics for column family are not reported
Yanfei Lei created FLINK-35495: -- Summary: The native metrics for column family are not reported Key: FLINK-35495 URL: https://issues.apache.org/jira/browse/FLINK-35495 Project: Flink Issue Type: Sub-task Reporter: Yanfei Lei -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35297][mysql] Add validation for option connect.timeout [flink-cdc]
Shawn-Hx commented on PR #3295: URL: https://github.com/apache/flink-cdc/pull/3295#issuecomment-2141143675 cc @leonardBang @PatrickRen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35363) FLIP-449: Reorganization of flink-connector-jdbc
[ https://issues.apache.org/jira/browse/FLINK-35363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850901#comment-17850901 ] Leonard Xu commented on FLINK-35363: Hey [~eskabetxe]， assigned to you. > FLIP-449: Reorganization of flink-connector-jdbc > > > Key: FLINK-35363 > URL: https://issues.apache.org/jira/browse/FLINK-35363 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: João Boto >Assignee: João Boto >Priority: Major > Labels: pull-request-available > > Described in: > [FLIP-449|https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35363) FLIP-449: Reorganization of flink-connector-jdbc
[ https://issues.apache.org/jira/browse/FLINK-35363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu reassigned FLINK-35363: -- Assignee: João Boto > FLIP-449: Reorganization of flink-connector-jdbc > > > Key: FLINK-35363 > URL: https://issues.apache.org/jira/browse/FLINK-35363 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: João Boto >Assignee: João Boto >Priority: Major > Labels: pull-request-available > > Described in: > [FLIP-449|https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35484) Flink document file had removed but website can access
[ https://issues.apache.org/jira/browse/FLINK-35484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850900#comment-17850900 ] Leonard Xu commented on FLINK-35484: thanks [~gongzhongqiang] for the confirm, cool! > Flink document file had removed but website can access > -- > > Key: FLINK-35484 > URL: https://issues.apache.org/jira/browse/FLINK-35484 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Fix For: 1.20.0 > > > Flink 1.18 document had remove document about DataSet : issue link > https://issues.apache.org/jira/browse/FLINK-32741. > But I can still access the link : > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [docs] Improve the readme and issue template [flink-cdc]
leonardBang commented on code in PR #3383: URL: https://github.com/apache/flink-cdc/pull/3383#discussion_r1621585404 ## .github/ISSUE_TEMPLATE/issue-notice.yml: ## @@ -47,6 +50,8 @@ body: 感谢您选择 Flink CDC! 由于 Apache Flink 的要求，请在 [Apache Jira](https://issues.apache.org/jira) 的 `Flink` 项目下使用 `Flink CDC` 标签来反馈缺陷或新功能。 +您需要登录 Jira 帐户才能创建 issue 去记录缺陷或新功能，如果您没有 Apache JIRA 帐户，您可以在 [ASF Jira 账号](https://selfserve.apache.org/jira-account.html) +申请账号，账号创建会由申请项目的 PMC member 审核，通常一到两个工作日可以通过。 Review Comment: 本意是想照顾下中文用户，特别是新用户，使用 “ASF自助服务门户” 会让中文新用户特别懵，所以还是“ ASF Jira 账号” 用户好理解 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [docs] Improve the readme and issue template [flink-cdc]
leonardBang commented on code in PR #3383: URL: https://github.com/apache/flink-cdc/pull/3383#discussion_r1621584793 ## .github/ISSUE_TEMPLATE/issue-notice.yml: ## @@ -47,6 +50,8 @@ body: 感谢您选择 Flink CDC! 由于 Apache Flink 的要求，请在 [Apache Jira](https://issues.apache.org/jira) 的 `Flink` 项目下使用 `Flink CDC` 标签来反馈缺陷或新功能。 +您需要登录 Jira 帐户才能创建 issue 去记录缺陷或新功能，如果您没有 Apache JIRA 帐户，您可以在 [ASF Jira 账号](https://selfserve.apache.org/jira-account.html) +申请账号，账号创建会由申请项目的 PMC member 审核，通常一到两个工作日可以通过。 Review Comment: selfserve.apache.org/jira-account == ASF Jira 账号 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35483) BatchJobRecoveryTest related to JM failover produced no output for 900 second
[ https://issues.apache.org/jira/browse/FLINK-35483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850894#comment-17850894 ] Weijie Guo commented on FLINK-35483: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59980&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=10345 > BatchJobRecoveryTest related to JM failover produced no output for 900 second > - > > Key: FLINK-35483 > URL: https://issues.apache.org/jira/browse/FLINK-35483 > Project: Flink > Issue Type: Bug > Components: Build System / CI >Affects Versions: 1.20.0 >Reporter: Weijie Guo >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > > testRecoverFromJMFailover > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59919&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9476 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
[ https://issues.apache.org/jira/browse/FLINK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850892#comment-17850892 ] Rui Fan commented on FLINK-33463: - After discuss with [~RocMarshal] , it's better to support the JDBC dynamic source tables based on the new source after we think new JdbcSource is stable, so let's revert the commit first, and finish the Jira in the future. > Support the implementation of dynamic source tables based on the new source > --- > > Key: FLINK-33463 > URL: https://issues.apache.org/jira/browse/FLINK-33463 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > Fix For: jdbc-3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33463][Connector/JDBC] Support the implementation of dynamic source tables based on the new source [flink-connector-jdbc]
1996fanrui commented on PR #117: URL: https://github.com/apache/flink-connector-jdbc/pull/117#issuecomment-2141086375 > Maybe we should revert this commit and re-commit it in the next second or third version Sounds make sense to me. The JdbcSource is introduced in the current version, it may be not stable. So it's better to consider it as the default Jdbc connector for SQL job after community think it's stable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-35484) Flink document file had removed but website can access
[ https://issues.apache.org/jira/browse/FLINK-35484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850890#comment-17850890 ] Zhongqiang Gong edited comment on FLINK-35484 at 5/31/24 1:31 AM: -- [~leonard] After build document ci scheduled，I checked: * The link: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ is 404 not foud * Document website is functioning properly. All result are excepted. was (Author: JIRAUSER301076): [~leonard] After build document ci scheduled，I checked: * The link: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ is 404 not foud * Document website is functioning properly. > Flink document file had removed but website can access > -- > > Key: FLINK-35484 > URL: https://issues.apache.org/jira/browse/FLINK-35484 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Fix For: 1.20.0 > > > Flink 1.18 document had remove document about DataSet : issue link > https://issues.apache.org/jira/browse/FLINK-32741. > But I can still access the link : > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35484) Flink document file had removed but website can access
[ https://issues.apache.org/jira/browse/FLINK-35484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850890#comment-17850890 ] Zhongqiang Gong edited comment on FLINK-35484 at 5/31/24 1:31 AM: -- [~leonard] After build document ci scheduled，I checked: * The link: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ is 404 not foud * Document website is functioning properly. was (Author: JIRAUSER301076): After build document ci scheduled，I checked: * The link: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ is 404 not foud * Document website is functioning properly. > Flink document file had removed but website can access > -- > > Key: FLINK-35484 > URL: https://issues.apache.org/jira/browse/FLINK-35484 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Fix For: 1.20.0 > > > Flink 1.18 document had remove document about DataSet : issue link > https://issues.apache.org/jira/browse/FLINK-32741. > But I can still access the link : > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35484) Flink document file had removed but website can access
[ https://issues.apache.org/jira/browse/FLINK-35484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850890#comment-17850890 ] Zhongqiang Gong commented on FLINK-35484: - After build document ci scheduled，I checked: * The link: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ is 404 not foud * Document website is functioning properly. > Flink document file had removed but website can access > -- > > Key: FLINK-35484 > URL: https://issues.apache.org/jira/browse/FLINK-35484 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Fix For: 1.20.0 > > > Flink 1.18 document had remove document about DataSet : issue link > https://issues.apache.org/jira/browse/FLINK-32741. > But I can still access the link : > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34613][cdc] Support recover from a specific savepoint file [flink-cdc]
monologuist commented on PR #2959:
URL: https://github.com/apache/flink-cdc/pull/2959#issuecomment-2141075378
> @lvyanquan hi，请问您这个PR，所预期的从savepoint重启Flink CDC Pipeline作业的规范命令是这样吗？
>
> ```
> 在${FLINK_CDC_HOME}目录下执行：
> $ ./bin/flink-cdc.sh mysql-to-doris.yaml -s
/tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab
> ```
>
> 因为我没有看到明确的重启命令说明*，所以对此怀有疑惑。并且我认为应当明确指出。
>
> *您引用的链接中，我看到的仍是针对jar作业的重启方式。 $ ./bin/flink run --detached \
--from-savepoint
/tmp/[flink-savepoints](https://issues.apache.org/jira/browse/FLINK-savepoints)/savepoint-cca7bc-bb1e257f0dab
./FlinkCDCExample.jar
经多次测试，我确认这个命令可行。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-25537] [JUnit5 Migration] Module: flink-core with,Package: core [flink]
GOODBOY008 commented on PR #24685: URL: https://github.com/apache/flink/pull/24685#issuecomment-2141075315 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-25537] [JUnit5 Migration] Module: flink-core with,Package: core [flink]
GOODBOY008 commented on PR #24685: URL: https://github.com/apache/flink/pull/24685#issuecomment-2140991378 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-34023) Expose Kinesis client retry config in sink
[
https://issues.apache.org/jira/browse/FLINK-34023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850841#comment-17850841
]
Brad Atcheson edited comment on FLINK-34023 at 5/30/24 7:28 PM:
[Here's|[https://github.com/brada/flink-connector-aws/commit/aa4fce309e94f979aa1f3ef48d5d27fa17f55759]]
a possible solution that adds 3 new config properties:
- aws.dynamodb.client.retry-policy.num-retries
- aws.firehose.client.retry-policy.num-retries
- aws.kinesis.client.retry-policy.num-retries
That seemed like the simplest possible approach. Exposing the complete AWS SDK
retry policy would require many more parameters, including [backoff
strategy|[https://sdk.amazonaws.com/java/api/2.0.0/software/amazon/awssdk/core/retry/backoff/BackoffStrategy.html]],
base delay, max backoff time etc. Since num-retries is the most important
parameter, would it be acceptable to expose just that one for now?
The `mvn clean verify` and `mvm clean package` commands fail on that branch,
but they also fail on the origin main branch for what appears to be issues
unrelated to my commit:
{noformat}
[ERROR] Failures:
[ERROR] RowDataToAttributeValueConverterTest.testFloat:208
Expecting map:
{"key"=AttributeValue(N=1.2345679E17)}
to contain entries:
["key"=AttributeValue(N=1.23456791E17)]
but the following map entries had different values:
["key"=AttributeValue(N=1.2345679E17) (expected:
AttributeValue(N=1.23456791E17))]{noformat}
was (Author: JIRAUSER303710):
[Here's|[https://github.com/brada/flink-connector-aws/commit/aa4fce309e94f979aa1f3ef48d5d27fa17f55759]]
a possible solution that adds 3 new config properties:
- aws.dynamodb.client.retry-policy.num-retries
- aws.firehose.client.retry-policy.num-retries
- aws.kinesis.client.retry-policy.num-retries
That seemed like the simplest possible approach. Exposing the complete AWS SDK
retry policy would require many more parameters, including [backoff
strategy|[https://sdk.amazonaws.com/java/api/2.0.0/software/amazon/awssdk/core/retry/backoff/BackoffStrategy.html]],
base delay, max backoff time etc. Since num-retries is the most important
parameter, would it be acceptable to expose just that one for now?
The `mvn clean verify` and `mvm clean package` commands fail on that branch,
but they also fail on the origin main branch for what appears to be issues
unrelated to my commit:
```
[ERROR] Failures:
[ERROR] RowDataToAttributeValueConverterTest.testFloat:208
Expecting map:
\{"key"=AttributeValue(N=1.2345679E17)}
to contain entries:
["key"=AttributeValue(N=1.23456791E17)]
but the following map entries had different values:
["key"=AttributeValue(N=1.2345679E17) (expected:
AttributeValue(N=1.23456791E17))]
```
> Expose Kinesis client retry config in sink
> --
>
> Key: FLINK-34023
> URL: https://issues.apache.org/jira/browse/FLINK-34023
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
>Reporter: Brad Atcheson
>Priority: Major
>
> The consumer side exposes client retry configuration like
> [flink.shard.getrecords.maxretries|https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.html#SHARD_GETRECORDS_RETRIES]
> but the producer side lacks similar config for PutRecords.
> The KinesisStreamsSinkWriter constructor calls
> {code}
> this.httpClient =
> AWSGeneralUtil.createAsyncHttpClient(kinesisClientProperties);
> this.kinesisClient = buildClient(kinesisClientProperties, this.httpClient);
> {code}
> But those methods only refer to these values (aside from
> endpoint/region/creds) in the kinesisClientProperties:
> * aws.http-client.max-concurrency
> * aws.http-client.read-timeout
> * aws.trust.all.certificates
> * aws.http.protocol.version
> Without control over retry, users can observe exceptions like {code}Request
> attempt 2 failure: Unable to execute HTTP request: connection timed out after
> 2000 ms: kinesis.us-west-2.amazonaws.com{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34023) Expose Kinesis client retry config in sink
[
https://issues.apache.org/jira/browse/FLINK-34023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850841#comment-17850841
]
Brad Atcheson edited comment on FLINK-34023 at 5/30/24 7:28 PM:
[Here's|https://github.com/brada/flink-connector-aws/commit/aa4fce309e94f979aa1f3ef48d5d27fa17f55759]
a possible solution that adds 3 new config properties:
- aws.dynamodb.client.retry-policy.num-retries
- aws.firehose.client.retry-policy.num-retries
- aws.kinesis.client.retry-policy.num-retries
That seemed like the simplest possible approach. Exposing the complete AWS SDK
retry policy would require many more parameters, including [backoff
strategy|https://sdk.amazonaws.com/java/api/2.0.0/software/amazon/awssdk/core/retry/backoff/BackoffStrategy.html],
base delay, max backoff time etc. Since num-retries is the most important
parameter, would it be acceptable to expose just that one for now?
The `mvn clean verify` and `mvm clean package` commands fail on that branch,
but they also fail on the origin main branch for what appears to be issues
unrelated to my commit:
{noformat}
[ERROR] Failures:
[ERROR] RowDataToAttributeValueConverterTest.testFloat:208
Expecting map:
{"key"=AttributeValue(N=1.2345679E17)}
to contain entries:
["key"=AttributeValue(N=1.23456791E17)]
but the following map entries had different values:
["key"=AttributeValue(N=1.2345679E17) (expected:
AttributeValue(N=1.23456791E17))]{noformat}
was (Author: JIRAUSER303710):
[Here's|[https://github.com/brada/flink-connector-aws/commit/aa4fce309e94f979aa1f3ef48d5d27fa17f55759]]
a possible solution that adds 3 new config properties:
- aws.dynamodb.client.retry-policy.num-retries
- aws.firehose.client.retry-policy.num-retries
- aws.kinesis.client.retry-policy.num-retries
That seemed like the simplest possible approach. Exposing the complete AWS SDK
retry policy would require many more parameters, including [backoff
strategy|[https://sdk.amazonaws.com/java/api/2.0.0/software/amazon/awssdk/core/retry/backoff/BackoffStrategy.html]],
base delay, max backoff time etc. Since num-retries is the most important
parameter, would it be acceptable to expose just that one for now?
The `mvn clean verify` and `mvm clean package` commands fail on that branch,
but they also fail on the origin main branch for what appears to be issues
unrelated to my commit:
{noformat}
[ERROR] Failures:
[ERROR] RowDataToAttributeValueConverterTest.testFloat:208
Expecting map:
{"key"=AttributeValue(N=1.2345679E17)}
to contain entries:
["key"=AttributeValue(N=1.23456791E17)]
but the following map entries had different values:
["key"=AttributeValue(N=1.2345679E17) (expected:
AttributeValue(N=1.23456791E17))]{noformat}
> Expose Kinesis client retry config in sink
> --
>
> Key: FLINK-34023
> URL: https://issues.apache.org/jira/browse/FLINK-34023
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
>Reporter: Brad Atcheson
>Priority: Major
>
> The consumer side exposes client retry configuration like
> [flink.shard.getrecords.maxretries|https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.html#SHARD_GETRECORDS_RETRIES]
> but the producer side lacks similar config for PutRecords.
> The KinesisStreamsSinkWriter constructor calls
> {code}
> this.httpClient =
> AWSGeneralUtil.createAsyncHttpClient(kinesisClientProperties);
> this.kinesisClient = buildClient(kinesisClientProperties, this.httpClient);
> {code}
> But those methods only refer to these values (aside from
> endpoint/region/creds) in the kinesisClientProperties:
> * aws.http-client.max-concurrency
> * aws.http-client.read-timeout
> * aws.trust.all.certificates
> * aws.http.protocol.version
> Without control over retry, users can observe exceptions like {code}Request
> attempt 2 failure: Unable to execute HTTP request: connection timed out after
> 2000 ms: kinesis.us-west-2.amazonaws.com{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34023) Expose Kinesis client retry config in sink
[
https://issues.apache.org/jira/browse/FLINK-34023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850841#comment-17850841
]
Brad Atcheson edited comment on FLINK-34023 at 5/30/24 7:26 PM:
[Here's|[https://github.com/brada/flink-connector-aws/commit/aa4fce309e94f979aa1f3ef48d5d27fa17f55759]]
a possible solution that adds 3 new config properties:
- aws.dynamodb.client.retry-policy.num-retries
- aws.firehose.client.retry-policy.num-retries
- aws.kinesis.client.retry-policy.num-retries
That seemed like the simplest possible approach. Exposing the complete AWS SDK
retry policy would require many more parameters, including [backoff
strategy|[https://sdk.amazonaws.com/java/api/2.0.0/software/amazon/awssdk/core/retry/backoff/BackoffStrategy.html]],
base delay, max backoff time etc. Since num-retries is the most important
parameter, would it be acceptable to expose just that one for now?
The `mvn clean verify` and `mvm clean package` commands fail on that branch,
but they also fail on the origin main branch for what appears to be issues
unrelated to my commit:
```
[ERROR] Failures:
[ERROR] RowDataToAttributeValueConverterTest.testFloat:208
Expecting map:
\{"key"=AttributeValue(N=1.2345679E17)}
to contain entries:
["key"=AttributeValue(N=1.23456791E17)]
but the following map entries had different values:
["key"=AttributeValue(N=1.2345679E17) (expected:
AttributeValue(N=1.23456791E17))]
```
was (Author: JIRAUSER303710):
[Here's](https://github.com/brada/flink-connector-aws/commit/aa4fce309e94f979aa1f3ef48d5d27fa17f55759)
a possible solution that adds 3 new config properties:
- aws.dynamodb.client.retry-policy.num-retries
- aws.firehose.client.retry-policy.num-retries
- aws.kinesis.client.retry-policy.num-retries
That seemed like the simplest possible approach. Exposing the complete AWS SDK
retry policy would require many more parameters, including [backoff
strategy](https://sdk.amazonaws.com/java/api/2.0.0/software/amazon/awssdk/core/retry/backoff/BackoffStrategy.html),
base delay, max backoff time etc. Since num-retries is the most important
parameter, would it be acceptable to expose just that one for now?
The `mvn clean verify` and `mvm clean package` commands fail on that branch,
but they also fail on the origin main branch for what appears to be issues
unrelated to my commit:
```
[ERROR] Failures:
[ERROR] RowDataToAttributeValueConverterTest.testFloat:208
Expecting map:
\{"key"=AttributeValue(N=1.2345679E17)}
to contain entries:
["key"=AttributeValue(N=1.23456791E17)]
but the following map entries had different values:
["key"=AttributeValue(N=1.2345679E17) (expected:
AttributeValue(N=1.23456791E17))]
```
> Expose Kinesis client retry config in sink
> --
>
> Key: FLINK-34023
> URL: https://issues.apache.org/jira/browse/FLINK-34023
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
>Reporter: Brad Atcheson
>Priority: Major
>
> The consumer side exposes client retry configuration like
> [flink.shard.getrecords.maxretries|https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.html#SHARD_GETRECORDS_RETRIES]
> but the producer side lacks similar config for PutRecords.
> The KinesisStreamsSinkWriter constructor calls
> {code}
> this.httpClient =
> AWSGeneralUtil.createAsyncHttpClient(kinesisClientProperties);
> this.kinesisClient = buildClient(kinesisClientProperties, this.httpClient);
> {code}
> But those methods only refer to these values (aside from
> endpoint/region/creds) in the kinesisClientProperties:
> * aws.http-client.max-concurrency
> * aws.http-client.read-timeout
> * aws.trust.all.certificates
> * aws.http.protocol.version
> Without control over retry, users can observe exceptions like {code}Request
> attempt 2 failure: Unable to execute HTTP request: connection timed out after
> 2000 ms: kinesis.us-west-2.amazonaws.com{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34023) Expose Kinesis client retry config in sink
[
https://issues.apache.org/jira/browse/FLINK-34023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850841#comment-17850841
]
Brad Atcheson commented on FLINK-34023:
---
[Here's](https://github.com/brada/flink-connector-aws/commit/aa4fce309e94f979aa1f3ef48d5d27fa17f55759)
a possible solution that adds 3 new config properties:
- aws.dynamodb.client.retry-policy.num-retries
- aws.firehose.client.retry-policy.num-retries
- aws.kinesis.client.retry-policy.num-retries
That seemed like the simplest possible approach. Exposing the complete AWS SDK
retry policy would require many more parameters, including [backoff
strategy](https://sdk.amazonaws.com/java/api/2.0.0/software/amazon/awssdk/core/retry/backoff/BackoffStrategy.html),
base delay, max backoff time etc. Since num-retries is the most important
parameter, would it be acceptable to expose just that one for now?
The `mvn clean verify` and `mvm clean package` commands fail on that branch,
but they also fail on the origin main branch for what appears to be issues
unrelated to my commit:
```
[ERROR] Failures:
[ERROR] RowDataToAttributeValueConverterTest.testFloat:208
Expecting map:
\{"key"=AttributeValue(N=1.2345679E17)}
to contain entries:
["key"=AttributeValue(N=1.23456791E17)]
but the following map entries had different values:
["key"=AttributeValue(N=1.2345679E17) (expected:
AttributeValue(N=1.23456791E17))]
```
> Expose Kinesis client retry config in sink
> --
>
> Key: FLINK-34023
> URL: https://issues.apache.org/jira/browse/FLINK-34023
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
>Reporter: Brad Atcheson
>Priority: Major
>
> The consumer side exposes client retry configuration like
> [flink.shard.getrecords.maxretries|https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.html#SHARD_GETRECORDS_RETRIES]
> but the producer side lacks similar config for PutRecords.
> The KinesisStreamsSinkWriter constructor calls
> {code}
> this.httpClient =
> AWSGeneralUtil.createAsyncHttpClient(kinesisClientProperties);
> this.kinesisClient = buildClient(kinesisClientProperties, this.httpClient);
> {code}
> But those methods only refer to these values (aside from
> endpoint/region/creds) in the kinesisClientProperties:
> * aws.http-client.max-concurrency
> * aws.http-client.read-timeout
> * aws.trust.all.certificates
> * aws.http.protocol.version
> Without control over retry, users can observe exceptions like {code}Request
> attempt 2 failure: Unable to execute HTTP request: connection timed out after
> 2000 ms: kinesis.us-west-2.amazonaws.com{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35472] Improve tests for Elasticsearch 8 connector [flink-connector-elasticsearch]
reta commented on code in PR #105:
URL:
https://github.com/apache/flink-connector-elasticsearch/pull/105#discussion_r1621068383
##
flink-connector-elasticsearch8/src/test/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchSinkBaseITCase.java:
##
@@ -21,86 +21,144 @@
package org.apache.flink.connector.elasticsearch.sink;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameter;
+import
org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
+
+import co.elastic.clients.elasticsearch.core.bulk.BulkOperationVariant;
+import co.elastic.clients.elasticsearch.core.bulk.IndexOperation;
import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.elasticsearch.ElasticsearchContainer;
-import org.testcontainers.junit.jupiter.Container;
-import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import java.io.IOException;
import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
-/** Base Integration tests class. */
-@Testcontainers
-public class ElasticsearchSinkBaseITCase {
+/**
+ * {@link ElasticsearchSinkBaseITCase} is the base class for integration tests.
+ *
+ * It is extended with the {@link ParameterizedTestExtension} for
parameterized testing against
+ * secure and non-secure Elasticsearch clusters. Tests must be annotated by
{@link TestTemplate} in
+ * order to be parameterized.
+ *
+ * The cluster is running via test containers. In order to reuse the
singleton containers by all
+ * inheriting test classes, we manage their lifecycle. The two containers are
started only once when
+ * this class is loaded. At the end of the test suite the Ryuk container that
is started by
+ * Testcontainers core will take care of stopping the singleton container.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
Review Comment:
Thank you, @liuml07 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35472] Improve tests for Elasticsearch 8 connector [flink-connector-elasticsearch]
liuml07 commented on code in PR #105:
URL:
https://github.com/apache/flink-connector-elasticsearch/pull/105#discussion_r1621057382
##
flink-connector-elasticsearch8/src/test/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchSinkBaseITCase.java:
##
@@ -21,86 +21,144 @@
package org.apache.flink.connector.elasticsearch.sink;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameter;
+import
org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
+
+import co.elastic.clients.elasticsearch.core.bulk.BulkOperationVariant;
+import co.elastic.clients.elasticsearch.core.bulk.IndexOperation;
import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.elasticsearch.ElasticsearchContainer;
-import org.testcontainers.junit.jupiter.Container;
-import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import java.io.IOException;
import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
-/** Base Integration tests class. */
-@Testcontainers
-public class ElasticsearchSinkBaseITCase {
+/**
+ * {@link ElasticsearchSinkBaseITCase} is the base class for integration tests.
+ *
+ * It is extended with the {@link ParameterizedTestExtension} for
parameterized testing against
+ * secure and non-secure Elasticsearch clusters. Tests must be annotated by
{@link TestTemplate} in
+ * order to be parameterized.
+ *
+ * The cluster is running via test containers. In order to reuse the
singleton containers by all
+ * inheriting test classes, we manage their lifecycle. The two containers are
started only once when
+ * this class is loaded. At the end of the test suite the Ryuk container that
is started by
+ * Testcontainers core will take care of stopping the singleton container.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
Review Comment:
Thank you very much for checking again!
This manual lifecycle management for singleton test containers is new to me
as well. I presume Ryuk's delay-based container cleanup is not straightforward
to many people, so I also put a comment in the base test class.
```
*
* The cluster is running via test containers. In order to reuse the
singleton containers by all
* inheriting test classes, we manage their lifecycle. The two containers
are started only once when
* this class is loaded. At the end of the test suite the Ryuk container
that is started by
* Testcontainers core will take care of stopping the singleton container.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-35494) Reorganize sources
João Boto created FLINK-35494: - Summary: Reorganize sources Key: FLINK-35494 URL: https://issues.apache.org/jira/browse/FLINK-35494 Project: Flink Issue Type: Sub-task Reporter: João Boto -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35472] Improve tests for Elasticsearch 8 connector [flink-connector-elasticsearch]
reta commented on code in PR #105:
URL:
https://github.com/apache/flink-connector-elasticsearch/pull/105#discussion_r1621010106
##
flink-connector-elasticsearch8/src/test/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchSinkBaseITCase.java:
##
@@ -21,86 +21,144 @@
package org.apache.flink.connector.elasticsearch.sink;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameter;
+import
org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
+
+import co.elastic.clients.elasticsearch.core.bulk.BulkOperationVariant;
+import co.elastic.clients.elasticsearch.core.bulk.IndexOperation;
import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.elasticsearch.ElasticsearchContainer;
-import org.testcontainers.junit.jupiter.Container;
-import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import java.io.IOException;
import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
-/** Base Integration tests class. */
-@Testcontainers
-public class ElasticsearchSinkBaseITCase {
+/**
+ * {@link ElasticsearchSinkBaseITCase} is the base class for integration tests.
+ *
+ * It is extended with the {@link ParameterizedTestExtension} for
parameterized testing against
+ * secure and non-secure Elasticsearch clusters. Tests must be annotated by
{@link TestTemplate} in
+ * order to be parameterized.
+ *
+ * The cluster is running via test containers. In order to reuse the
singleton containers by all
+ * inheriting test classes, we manage their lifecycle. The two containers are
started only once when
+ * this class is loaded. At the end of the test suite the Ryuk container that
is started by
+ * Testcontainers core will take care of stopping the singleton container.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
Review Comment:
> The idea of using ParameterizedTestExtension is to make all subclasses run
with both secure and non-secure tests by default. So tests do not actually
depend on one static container, but instead use both of them.
Oh, apologies, missed that, I think the approach makes perfect sense that.
> I have been reading the doc of testcontainers and didn't find a solution
that works best for base classes via @Container managed lifecycle and still
efficiently.
Got it, thanks for reference but we still have a problem here since
containers are not shutdown properly (we do start them up), that the concerns I
have vs using `@Container`, let me try to look into that.
UPD: seems to be covered, from the docs you referenced [1]:
> At the end of the test suite the [Ryuk
container](https://github.com/testcontainers/moby-ryuk) that is started by
Testcontainers core will take care of stopping the singleton container.
[1]
https://java.testcontainers.org/test_framework_integration/manual_lifecycle_control/#singleton-containers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35472] Improve tests for Elasticsearch 8 connector [flink-connector-elasticsearch]
reta commented on code in PR #105:
URL:
https://github.com/apache/flink-connector-elasticsearch/pull/105#discussion_r1621010106
##
flink-connector-elasticsearch8/src/test/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchSinkBaseITCase.java:
##
@@ -21,86 +21,144 @@
package org.apache.flink.connector.elasticsearch.sink;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameter;
+import
org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
+
+import co.elastic.clients.elasticsearch.core.bulk.BulkOperationVariant;
+import co.elastic.clients.elasticsearch.core.bulk.IndexOperation;
import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.elasticsearch.ElasticsearchContainer;
-import org.testcontainers.junit.jupiter.Container;
-import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import java.io.IOException;
import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
-/** Base Integration tests class. */
-@Testcontainers
-public class ElasticsearchSinkBaseITCase {
+/**
+ * {@link ElasticsearchSinkBaseITCase} is the base class for integration tests.
+ *
+ * It is extended with the {@link ParameterizedTestExtension} for
parameterized testing against
+ * secure and non-secure Elasticsearch clusters. Tests must be annotated by
{@link TestTemplate} in
+ * order to be parameterized.
+ *
+ * The cluster is running via test containers. In order to reuse the
singleton containers by all
+ * inheriting test classes, we manage their lifecycle. The two containers are
started only once when
+ * this class is loaded. At the end of the test suite the Ryuk container that
is started by
+ * Testcontainers core will take care of stopping the singleton container.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
Review Comment:
> The idea of using ParameterizedTestExtension is to make all subclasses run
with both secure and non-secure tests by default. So tests do not actually
depend on one static container, but instead use both of them.
Oh, apologies, missed that, I think the approach makes perfect sense that.
> I have been reading the doc of testcontainers and didn't find a solution
that works best for base classes via @Container managed lifecycle and still
efficiently.
Got it, thank for reference but we still have a problem here since
containers are not shutdown properly (we do start them up), that the concerns I
have vs using `@Container`, let me try to look into that.
##
flink-connector-elasticsearch8/src/test/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchSinkBaseITCase.java:
##
@@ -21,86 +21,144 @@
package org.apache.flink.connector.elasticsearch.sink;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameter;
+import
org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
+
+import co.elastic.clients.elasticsearch.core.bulk.BulkOperationVariant;
+import co.elastic.clients.elasticsearch.core.bulk.IndexOperation;
import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontai
Re: [PR] [FLINK-35483][runtime] Fix unstable BatchJobRecoveryTest. [flink]
zhuzhurk commented on code in PR #24871:
URL: https://github.com/apache/flink/pull/24871#discussion_r1620899559
##
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java:
##
@@ -257,6 +260,11 @@ void testRecoverFromJMFailover() throws Exception {
createScheduler(deserializeJobGraph(serializedJobGraph));
startSchedulingAndWaitRecoverFinish(newScheduler);
+waitUntilExecutionVertexState(
Review Comment:
Why is it needed?
IIUC, the status of source vertices should have recovered before the
recovery process finishes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [hotfix][docs] Fix typo in alter.md [flink]
potterxu commented on PR #24858: URL: https://github.com/apache/flink/pull/24858#issuecomment-2139841619 Hi @swuferhong @luoyuxia , would you mind take a look at the above changes in doc? Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35399) Add documents for batch job master failure recovery
[ https://issues.apache.org/jira/browse/FLINK-35399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850739#comment-17850739 ] Zhu Zhu commented on FLINK-35399: - 56cd9607713d0da874dcc54c4cf6d5b3b52b1050 refined the doc a bit. > Add documents for batch job master failure recovery > --- > > Key: FLINK-35399 > URL: https://issues.apache.org/jira/browse/FLINK-35399 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Zhu Zhu >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35399][doc] Update the batch job recovery documentation. [flink]
zhuzhurk merged PR #24872: URL: https://github.com/apache/flink/pull/24872 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32384) Remove deprecated configuration keys which violate YAML spec
[ https://issues.apache.org/jira/browse/FLINK-32384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850734#comment-17850734 ] Zhu Zhu commented on FLINK-32384: - Thanks for volunteering to contribute to Flink. [~kartikeypant] However, this is a breaking change. Therefore, we cannot do it until Flink 1.20 is released and release cycle of Flink 2.0 is started. You are welcome to take this task if you are free at that moment. Before that, you can take a look at other tickets. > Remove deprecated configuration keys which violate YAML spec > > > Key: FLINK-32384 > URL: https://issues.apache.org/jira/browse/FLINK-32384 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Reporter: Zhu Zhu >Priority: Major > Labels: 2.0-related > Fix For: 2.0.0 > > > In FLINK-29372, key that violate YAML spec are renamed to a valid form and > the old names are deprecated. > In Flink 2.0 we should remove these deprecated keys. This will prevent users > (unintentionally) to create invalid YAML form flink-conf.yaml. > Then with the work of FLINK-23620, we can remove the non-standard YAML > parsing logic and enforce standard YAML validation in CI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34746) Switching to the Apache CDN for Dockerfile
[
https://issues.apache.org/jira/browse/FLINK-34746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850732#comment-17850732
]
Hong Liang Teoh commented on FLINK-34746:
-
merged commit
[{{7f63237}}|https://github.com/apache/flink-docker/commit/7f63237615138615826f2820ca54ff2054514fca]
into apache:dev-1.19
merged commit
[{{804c9f3}}|https://github.com/apache/flink-docker/commit/804c9f3bb6772751d09252b6d15e8a1aac4ca055]
into apache:dev-1.18
merged commit
[{{0ac313e}}|https://github.com/apache/flink-docker/commit/0ac313e39fda6c49778fbdf15f4b5d827476253a]
into apache:dev-1.17
> Switching to the Apache CDN for Dockerfile
> --
>
> Key: FLINK-34746
> URL: https://issues.apache.org/jira/browse/FLINK-34746
> Project: Flink
> Issue Type: Improvement
> Components: flink-docker
>Reporter: lincoln lee
>Assignee: Martijn Visser
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0, 1.19.1
>
>
> During publishing the official image, we received some comments
> for Switching to the Apache CDN
>
> See
> https://github.com/docker-library/official-images/pull/16114
> https://github.com/docker-library/official-images/pull/16430
>
> Reason for switching: [https://apache.org/history/mirror-history.html] (also
> [https://www.apache.org/dyn/closer.cgi] and [https://www.apache.org/mirrors])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34746] Switching to the Apache CDN for Dockerfile [flink-docker]
hlteoh37 merged PR #193: URL: https://github.com/apache/flink-docker/pull/193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34746] Switching to the Apache CDN for Dockerfile [flink-docker]
hlteoh37 merged PR #192: URL: https://github.com/apache/flink-docker/pull/192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34746] Switching to the Apache CDN for Dockerfile [flink-docker]
hlteoh37 merged PR #191: URL: https://github.com/apache/flink-docker/pull/191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34746] Switching to the Apache CDN for Dockerfile [flink-docker]
hlteoh37 commented on PR #191: URL: https://github.com/apache/flink-docker/pull/191#issuecomment-2139676975 Thanks @MartijnVisser. Merging the PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34379) table.optimizer.dynamic-filtering.enabled lead to OutOfMemoryError
[ https://issues.apache.org/jira/browse/FLINK-34379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850730#comment-17850730 ] Hong Liang Teoh commented on FLINK-34379: - Closing Jira as patch has been completed for master branch as well as 1.17, 1.18, 1.19 branch > table.optimizer.dynamic-filtering.enabled lead to OutOfMemoryError > -- > > Key: FLINK-34379 > URL: https://issues.apache.org/jira/browse/FLINK-34379 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.17.2, 1.18.1 > Environment: 1.17.1 >Reporter: zhu >Assignee: Jeyhun Karimov >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.3, 1.18.2, 1.20.0, 1.19.1 > > > When using batch computing, I union all about 50 tables and then join other > table. When compiling the execution plan, > there throws OutOfMemoryError: Java heap space, which was no problem in > 1.15.2. However, both 1.17.2 and 1.18.1 all throws same errors,This causes > jobmanager to restart. Currently,it has been found that this is caused by > table.optimizer.dynamic-filtering.enabled, which defaults is true,When I set > table.optimizer.dynamic-filtering.enabled to false, it can be compiled and > executed normally > code > TableEnvironment.create(EnvironmentSettings.newInstance() > .withConfiguration(configuration) > .inBatchMode().build()) > sql=select att,filename,'table0' as mo_name from table0 UNION All select > att,filename,'table1' as mo_name from table1 UNION All select > att,filename,'table2' as mo_name from table2 UNION All select > att,filename,'table3' as mo_name from table3 UNION All select > att,filename,'table4' as mo_name from table4 UNION All select > att,filename,'table5' as mo_name from table5 UNION All select > att,filename,'table6' as mo_name from table6 UNION All select > att,filename,'table7' as mo_name from table7 UNION All select > att,filename,'table8' as mo_name from table8 UNION All select > att,filename,'table9' as mo_name from table9 UNION All select > att,filename,'table10' as mo_name from table10 UNION All select > att,filename,'table11' as mo_name from table11 UNION All select > att,filename,'table12' as mo_name from table12 UNION All select > att,filename,'table13' as mo_name from table13 UNION All select > att,filename,'table14' as mo_name from table14 UNION All select > att,filename,'table15' as mo_name from table15 UNION All select > att,filename,'table16' as mo_name from table16 UNION All select > att,filename,'table17' as mo_name from table17 UNION All select > att,filename,'table18' as mo_name from table18 UNION All select > att,filename,'table19' as mo_name from table19 UNION All select > att,filename,'table20' as mo_name from table20 UNION All select > att,filename,'table21' as mo_name from table21 UNION All select > att,filename,'table22' as mo_name from table22 UNION All select > att,filename,'table23' as mo_name from table23 UNION All select > att,filename,'table24' as mo_name from table24 UNION All select > att,filename,'table25' as mo_name from table25 UNION All select > att,filename,'table26' as mo_name from table26 UNION All select > att,filename,'table27' as mo_name from table27 UNION All select > att,filename,'table28' as mo_name from table28 UNION All select > att,filename,'table29' as mo_name from table29 UNION All select > att,filename,'table30' as mo_name from table30 UNION All select > att,filename,'table31' as mo_name from table31 UNION All select > att,filename,'table32' as mo_name from table32 UNION All select > att,filename,'table33' as mo_name from table33 UNION All select > att,filename,'table34' as mo_name from table34 UNION All select > att,filename,'table35' as mo_name from table35 UNION All select > att,filename,'table36' as mo_name from table36 UNION All select > att,filename,'table37' as mo_name from table37 UNION All select > att,filename,'table38' as mo_name from table38 UNION All select > att,filename,'table39' as mo_name from table39 UNION All select > att,filename,'table40' as mo_name from table40 UNION All select > att,filename,'table41' as mo_name from table41 UNION All select > att,filename,'table42' as mo_name from table42 UNION All select > att,filename,'table43' as mo_name from table43 UNION All select > att,filename,'table44' as mo_name from table44 UNION All select > att,filename,'table45' as mo_name from table45 UNION All select > att,filename,'table46' as mo_name from table46 UNION All select > att,filename,'table47' as mo_name from table47 UNION All select > att,filename,'table48' as mo_name from table48 UNION All select > att,filename,'table49' as mo_name from table49 UNION All select > att,filename,'table5
[jira] [Resolved] (FLINK-35358) Breaking change when loading artifacts
[
https://issues.apache.org/jira/browse/FLINK-35358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh resolved FLINK-35358.
-
Resolution: Fixed
> Breaking change when loading artifacts
> --
>
> Key: FLINK-35358
> URL: https://issues.apache.org/jira/browse/FLINK-35358
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, flink-docker
>Affects Versions: 1.19.0
>Reporter: Rasmus Thygesen
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: 1.19.1
>
>
> We have been using the following code snippet in our Dockerfiles for running
> a Flink job in application mode
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar
> /opt/flink/usrlib/artifacts/my-job.jar
> USER flink {code}
>
> Which has been working since at least around Flink 1.14, but the 1.19 update
> has broken our Dockerfiles. The fix is to put the jar file a step further out
> so the code snippet becomes
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
> USER flink {code}
>
> We have not spent too much time looking into what the cause is, but we get
> the stack trace
>
> {code:java}
> myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load
> the provided entrypoint class.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
> [flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The program's
> entry point class 'my.company.job.MyJob' was not found in the jar file.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | ... 4 more
> myjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
> my.company.job.MyJob
> myjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown
> Source) ~[?:?]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.Class.forName0(Native Method) ~[?:?]
> myjob-jobmanager-1 | at java.lang.Class.forName(Unknown Source) ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:479)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myj
[jira] [Closed] (FLINK-34379) table.optimizer.dynamic-filtering.enabled lead to OutOfMemoryError
[ https://issues.apache.org/jira/browse/FLINK-34379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh closed FLINK-34379. --- Resolution: Fixed > table.optimizer.dynamic-filtering.enabled lead to OutOfMemoryError > -- > > Key: FLINK-34379 > URL: https://issues.apache.org/jira/browse/FLINK-34379 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.17.2, 1.18.1 > Environment: 1.17.1 >Reporter: zhu >Assignee: Jeyhun Karimov >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.3, 1.18.2, 1.20.0, 1.19.1 > > > When using batch computing, I union all about 50 tables and then join other > table. When compiling the execution plan, > there throws OutOfMemoryError: Java heap space, which was no problem in > 1.15.2. However, both 1.17.2 and 1.18.1 all throws same errors,This causes > jobmanager to restart. Currently,it has been found that this is caused by > table.optimizer.dynamic-filtering.enabled, which defaults is true,When I set > table.optimizer.dynamic-filtering.enabled to false, it can be compiled and > executed normally > code > TableEnvironment.create(EnvironmentSettings.newInstance() > .withConfiguration(configuration) > .inBatchMode().build()) > sql=select att,filename,'table0' as mo_name from table0 UNION All select > att,filename,'table1' as mo_name from table1 UNION All select > att,filename,'table2' as mo_name from table2 UNION All select > att,filename,'table3' as mo_name from table3 UNION All select > att,filename,'table4' as mo_name from table4 UNION All select > att,filename,'table5' as mo_name from table5 UNION All select > att,filename,'table6' as mo_name from table6 UNION All select > att,filename,'table7' as mo_name from table7 UNION All select > att,filename,'table8' as mo_name from table8 UNION All select > att,filename,'table9' as mo_name from table9 UNION All select > att,filename,'table10' as mo_name from table10 UNION All select > att,filename,'table11' as mo_name from table11 UNION All select > att,filename,'table12' as mo_name from table12 UNION All select > att,filename,'table13' as mo_name from table13 UNION All select > att,filename,'table14' as mo_name from table14 UNION All select > att,filename,'table15' as mo_name from table15 UNION All select > att,filename,'table16' as mo_name from table16 UNION All select > att,filename,'table17' as mo_name from table17 UNION All select > att,filename,'table18' as mo_name from table18 UNION All select > att,filename,'table19' as mo_name from table19 UNION All select > att,filename,'table20' as mo_name from table20 UNION All select > att,filename,'table21' as mo_name from table21 UNION All select > att,filename,'table22' as mo_name from table22 UNION All select > att,filename,'table23' as mo_name from table23 UNION All select > att,filename,'table24' as mo_name from table24 UNION All select > att,filename,'table25' as mo_name from table25 UNION All select > att,filename,'table26' as mo_name from table26 UNION All select > att,filename,'table27' as mo_name from table27 UNION All select > att,filename,'table28' as mo_name from table28 UNION All select > att,filename,'table29' as mo_name from table29 UNION All select > att,filename,'table30' as mo_name from table30 UNION All select > att,filename,'table31' as mo_name from table31 UNION All select > att,filename,'table32' as mo_name from table32 UNION All select > att,filename,'table33' as mo_name from table33 UNION All select > att,filename,'table34' as mo_name from table34 UNION All select > att,filename,'table35' as mo_name from table35 UNION All select > att,filename,'table36' as mo_name from table36 UNION All select > att,filename,'table37' as mo_name from table37 UNION All select > att,filename,'table38' as mo_name from table38 UNION All select > att,filename,'table39' as mo_name from table39 UNION All select > att,filename,'table40' as mo_name from table40 UNION All select > att,filename,'table41' as mo_name from table41 UNION All select > att,filename,'table42' as mo_name from table42 UNION All select > att,filename,'table43' as mo_name from table43 UNION All select > att,filename,'table44' as mo_name from table44 UNION All select > att,filename,'table45' as mo_name from table45 UNION All select > att,filename,'table46' as mo_name from table46 UNION All select > att,filename,'table47' as mo_name from table47 UNION All select > att,filename,'table48' as mo_name from table48 UNION All select > att,filename,'table49' as mo_name from table49 UNION All select > att,filename,'table50' as mo_name from table50 UNION All select > att,filename,'table51' as mo_name from table51 UNION All select > att,filename,'table
[jira] [Updated] (FLINK-35367) Reorganize sinks
[ https://issues.apache.org/jira/browse/FLINK-35367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] João Boto updated FLINK-35367: -- Summary: Reorganize sinks (was: Reorganize datastream sink and source) > Reorganize sinks > > > Key: FLINK-35367 > URL: https://issues.apache.org/jira/browse/FLINK-35367 > Project: Flink > Issue Type: Sub-task >Reporter: João Boto >Priority: Major > > Reorganize datastream sink and source -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35358) Breaking change when loading artifacts
[
https://issues.apache.org/jira/browse/FLINK-35358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh updated FLINK-35358:
Fix Version/s: 1.20.0
> Breaking change when loading artifacts
> --
>
> Key: FLINK-35358
> URL: https://issues.apache.org/jira/browse/FLINK-35358
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, flink-docker
>Affects Versions: 1.19.0
>Reporter: Rasmus Thygesen
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: 1.20.0, 1.19.1
>
>
> We have been using the following code snippet in our Dockerfiles for running
> a Flink job in application mode
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar
> /opt/flink/usrlib/artifacts/my-job.jar
> USER flink {code}
>
> Which has been working since at least around Flink 1.14, but the 1.19 update
> has broken our Dockerfiles. The fix is to put the jar file a step further out
> so the code snippet becomes
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
> USER flink {code}
>
> We have not spent too much time looking into what the cause is, but we get
> the stack trace
>
> {code:java}
> myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load
> the provided entrypoint class.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
> [flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The program's
> entry point class 'my.company.job.MyJob' was not found in the jar file.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | ... 4 more
> myjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
> my.company.job.MyJob
> myjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown
> Source) ~[?:?]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.Class.forName0(Native Method) ~[?:?]
> myjob-jobmanager-1 | at java.lang.Class.forName(Unknown Source) ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:479)
> ~[flink-dist-1.19.0.jar:1.1
[jira] [Commented] (FLINK-35358) Breaking change when loading artifacts
[
https://issues.apache.org/jira/browse/FLINK-35358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850729#comment-17850729
]
Hong Liang Teoh commented on FLINK-35358:
-
merged commit
[{{90a71a1}}|https://github.com/apache/flink/commit/90a71a124771697a0b8b2c2bbc520856d6ae9e25]
into apache:release-1.19
> Breaking change when loading artifacts
> --
>
> Key: FLINK-35358
> URL: https://issues.apache.org/jira/browse/FLINK-35358
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, flink-docker
>Affects Versions: 1.19.0
>Reporter: Rasmus Thygesen
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: 1.19.1
>
>
> We have been using the following code snippet in our Dockerfiles for running
> a Flink job in application mode
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar
> /opt/flink/usrlib/artifacts/my-job.jar
> USER flink {code}
>
> Which has been working since at least around Flink 1.14, but the 1.19 update
> has broken our Dockerfiles. The fix is to put the jar file a step further out
> so the code snippet becomes
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
> USER flink {code}
>
> We have not spent too much time looking into what the cause is, but we get
> the stack trace
>
> {code:java}
> myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load
> the provided entrypoint class.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
> [flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The program's
> entry point class 'my.company.job.MyJob' was not found in the jar file.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | ... 4 more
> myjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
> my.company.job.MyJob
> myjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown
> Source) ~[?:?]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.Class.forName0(Native Method) ~[?:?]
> myjob-jobmanager-1 | at java.lang.Class.forName(Unknown Source)
[jira] [Commented] (FLINK-35358) Breaking change when loading artifacts
[
https://issues.apache.org/jira/browse/FLINK-35358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850727#comment-17850727
]
Hong Liang Teoh commented on FLINK-35358:
-
merged commit
[{{853989b}}|https://github.com/apache/flink/commit/853989bd862c31e0c74cd5a584177dc401c5a3d4]
into apache:master
> Breaking change when loading artifacts
> --
>
> Key: FLINK-35358
> URL: https://issues.apache.org/jira/browse/FLINK-35358
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, flink-docker
>Affects Versions: 1.19.0
>Reporter: Rasmus Thygesen
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: 1.19.1
>
>
> We have been using the following code snippet in our Dockerfiles for running
> a Flink job in application mode
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar
> /opt/flink/usrlib/artifacts/my-job.jar
> USER flink {code}
>
> Which has been working since at least around Flink 1.14, but the 1.19 update
> has broken our Dockerfiles. The fix is to put the jar file a step further out
> so the code snippet becomes
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
> USER flink {code}
>
> We have not spent too much time looking into what the cause is, but we get
> the stack trace
>
> {code:java}
> myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load
> the provided entrypoint class.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
> [flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The program's
> entry point class 'my.company.job.MyJob' was not found in the jar file.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | ... 4 more
> myjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
> my.company.job.MyJob
> myjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown
> Source) ~[?:?]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.Class.forName0(Native Method) ~[?:?]
> myjob-jobmanager-1 | at java.lang.Class.forName(Unknown Source) ~[?:
Re: [PR] [FLINK-35358][clients] Reintroduce recursive JAR listing in classpath load from "usrlib" [flink]
hlteoh37 merged PR #24791: URL: https://github.com/apache/flink/pull/24791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.19][FLINK-35358][clients] Reintroduce recursive JAR listing in classpath load from "usrlib" [flink]
hlteoh37 merged PR #24792: URL: https://github.com/apache/flink/pull/24792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34746] Switching to the Apache CDN for Dockerfile [flink-docker]
hlteoh37 commented on PR #190: URL: https://github.com/apache/flink-docker/pull/190#issuecomment-2139650422 Appreciate if I could get your review please! @MartijnVisser or @lincoln-lil :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34746] Switching to the Apache CDN for Dockerfile [flink-docker]
hlteoh37 commented on PR #190: URL: https://github.com/apache/flink-docker/pull/190#issuecomment-2139649509 Backport for Flink 1.19: https://github.com/apache/flink-docker/pull/191 Backport for Flink 1.18: https://github.com/apache/flink-docker/pull/192 Backport for Flink 1.17: https://github.com/apache/flink-docker/pull/193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-26808][1.19] Limit FileUploadHandler to multipart routes [flink]
uce merged PR #24859: URL: https://github.com/apache/flink/pull/24859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35448] Translate pod templates documentation into Chinese [flink-kubernetes-operator]
RocMarshal commented on code in PR #830:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/830#discussion_r1620723423
##
docs/content/docs/custom-resource/pod-template.md:
##
@@ -113,10 +112,10 @@ arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
merged: [{name: a, p1: v1, p2: v2}, {name: c, p1: v1, p2: v2}]
```
-The operator supports an alternative array merging mechanism that can be
enabled by the `kubernetes.operator.pod-template.merge-arrays-by-name` flag.
-When true, instead of the default positional merging, object array elements
that have a `name` property defined will be merged by their name and the
resulting array will be a union of the two input arrays.
+Operator支持另一种数组合并机制，可以通过`kubernetes.operator.pod-template.merge-arrays-by-name`标志启用。
Review Comment:
```suggestion
Operator 支持另一种数组合并机制，可以通过
`kubernetes.operator.pod-template.merge-arrays-by-name` 标志启用。
```
##
docs/content/docs/custom-resource/pod-template.md:
##
@@ -93,16 +90,18 @@ spec:
```
{{< hint info >}}
-When using the operator with Flink native Kubernetes integration, please refer
to [pod template field precedence](
-https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink).
+当使用具有 Flink 原生 Kubernetes 集成的 operator 时，请参考 [pod template 字段优先级](
+https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink)。
{{< /hint >}}
+
## Array Merging Behaviour
-When layering pod templates (defining both a top level and jobmanager specific
podtemplate for example) the corresponding yamls are merged together.
+
+
+当分层 pod templates（例如同时定义顶级和任务管理器特定的 pod 模板）时，相应的 yaml 会合并在一起。
-The default behaviour of the pod template mechanism is to merge array arrays
by merging the objects in the respective array positions.
-This requires that containers in the podTemplates are defined in the same
order otherwise the results may be undefined.
+Pod 模板机制的默认行为是通过合并相应数组位置的对象合并数组类型的数组。
Review Comment:
`数组类型的数组` -> ` json 类型的数组`
Sorry for the previous comment on ignoring the demo lines.
Maybe the new comments is appropriate.
##
docs/content/docs/custom-resource/pod-template.md:
##
@@ -113,10 +112,10 @@ arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
merged: [{name: a, p1: v1, p2: v2}, {name: c, p1: v1, p2: v2}]
```
-The operator supports an alternative array merging mechanism that can be
enabled by the `kubernetes.operator.pod-template.merge-arrays-by-name` flag.
-When true, instead of the default positional merging, object array elements
that have a `name` property defined will be merged by their name and the
resulting array will be a union of the two input arrays.
+Operator支持另一种数组合并机制，可以通过`kubernetes.operator.pod-template.merge-arrays-by-name`标志启用。
+当为true时，将按名称合并具有`name`属性定义的对象数组元素，并且生成的数组将是两个输入数组的并集。
Review Comment:
```suggestion
当为 true 时，将按名称合并具有 `name` 属性定义的对象数组元素，并且生成的数组将是两个输入数组的并集。
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33463][Connector/JDBC] Support the implementation of dynamic source tables based on the new source [flink-connector-jdbc]
RocMarshal commented on PR #117: URL: https://github.com/apache/flink-connector-jdbc/pull/117#issuecomment-2139538600 Hi, @1996fanrui . Kafka introduced KafkaSource API in the Flink 1.12 branch https://issues.apache.org/jira/browse/FLINK-18323 Afterwards, in Flink version 1.14, KafkaDynamicTableSource was implemented based on KafkaSource https://issues.apache.org/jira/browse/FLINK-22914 So, Maybe we should revert this commit. Please let me know what's your opinion. CC @eskabetxe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33001) KafkaSource in batch mode failing with exception if topic partition is empty
[
https://issues.apache.org/jira/browse/FLINK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850678#comment-17850678
]
Naci Simsek edited comment on FLINK-33001 at 5/30/24 1:07 PM:
--
Just be able to try with the *flink-connector-kafka 3.1.0-1.18* on *Flink
1.18.1* session cluster with a batch job, and the exception is not there
anymore. The job finishes gracefully, without setting any specific log level
for any subclasses. I have the following log setting:
{code:java}
rootLogger.level = TRACE {code}
Therefore, the issue seems to be fixed.
However, there is another issue when setting a specific offset to start reading
from, which causes an offset reset and infitinite loop of FetchTasks, so that
the batch job keeps running, till there is an event received into the topic,
but of course this should be a content of another ticket.
was (Author: JIRAUSER302646):
Just be able to try with the *flink-connector-kafka 3.1.0-1.18* on *Flink
1.18.1* session cluster with a batch job, and the exception is not there
anymore. The job finishes gracefully, without setting any specific log level
for any subclasses. I have the following log setting:
{code:java}
rootLogger.level = TRACE {code}
Therefore, the issue seems to be fixed.
However, there is another issue when setting a specific offset to start reading
from, which causes an offset reset and endless loop of FetchTasks, so that the
batch job keeps running, till there is an event received into the topic, but of
course this should be a content of another ticket.
> KafkaSource in batch mode failing with exception if topic partition is empty
>
>
> Key: FLINK-33001
> URL: https://issues.apache.org/jira/browse/FLINK-33001
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.7, 1.14.6, 1.17.1
>Reporter: Abdul
>Priority: Major
>
> If the Kafka topic is empty in Batch mode, there is an exception while
> processing it. This bug was supposedly fixed but unfortunately, the exception
> still occurs. The original bug was reported as this
> https://issues.apache.org/jira/browse/FLINK-27041
> We tried to backport it but it still doesn't work.
> * The problem will occur in case of the DEBUG level of logger for class
> org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader
> * The same problems will occur in other versions of Flink, at least in the
> 1.15 release branch and tag release-1.15.4
> * The same problem also occurs in Flink 1.17.1 and 1.14
>
> The minimal code to produce this is
>
> {code:java}
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(RuntimeExecutionMode.BATCH);
> KafkaSource kafkaSource = KafkaSource
> .builder()
> .setBootstrapServers("localhost:9092")
> .setTopics("test_topic")
> .setValueOnlyDeserializer(new
> SimpleStringSchema())
> .setBounded(OffsetsInitializer.latest())
> .build();
> DataStream stream = env.fromSource(
> kafkaSource,
> WatermarkStrategy.noWatermarks(),
> "Kafka Source"
> );
> stream.print();
> env.execute("Flink KafkaSource test job"); {code}
> This produces exception:
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:199)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:154)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:116)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:275)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:67)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:398)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619)
> at
> org.apache.flink.streaming.runtime.tasks
Re: [PR] Enable async profiler [flink-benchmarks]
Zakelly commented on PR #90: URL: https://github.com/apache/flink-benchmarks/pull/90#issuecomment-2139458543 > @Zakelly I've pushed [decdc8e](https://github.com/apache/flink-benchmarks/commit/decdc8e2d161584a798461b4f381adba7cee731f) which restores the JMH defaults from 1.19. Hopefully that addresses the issues with the runtime under Jenkins. I'd just as happily revert that commit but it de-couples this PR from reasoning about the implications for changing the warmup and measurement times. I see. Thanks for your investigation. I'll give it a try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35199][table] Support the execution of create materialized table in full refresh mode [flink]
lsyldliu commented on code in PR #24866:
URL: https://github.com/apache/flink/pull/24866#discussion_r1620385497
##
flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManager.java:
##
@@ -640,10 +750,15 @@ private static byte[]
serializeContinuousHandler(ContinuousRefreshHandler refres
}
}
-private static ResolvedCatalogMaterializedTable
getCatalogMaterializedTable(
+private ResolvedCatalogMaterializedTable getCatalogMaterializedTable(
OperationExecutor operationExecutor, ObjectIdentifier
tableIdentifier) {
ResolvedCatalogBaseTable resolvedCatalogBaseTable =
-operationExecutor.getTable(tableIdentifier);
+operationExecutor
Review Comment:
Why do you need to change this code snippet? Original logical can't satisfy
your needs?
##
flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManager.java:
##
@@ -165,7 +215,73 @@ private static void createMaterializedInContinuousMode(
}
}
-private static ResultFetcher callAlterMaterializedTableSuspend(
+private void createMaterializedInFullMode(
+OperationExecutor operationExecutor,
+OperationHandle handle,
+CreateMaterializedTableOperation createMaterializedTableOperation)
{
+if (workflowScheduler == null) {
+throw new SqlExecutionException(
+"Scheduler is not configured, can't create materialized
table in full mode.");
+}
+// create materialized table first
+operationExecutor.callExecutableOperation(handle,
createMaterializedTableOperation);
+
+ObjectIdentifier materializedTableIdentifier =
+createMaterializedTableOperation.getTableIdentifier();
+CatalogMaterializedTable catalogMaterializedTable =
+createMaterializedTableOperation.getCatalogMaterializedTable();
+
+// convert duration to cron expression
+String cronExpression =
+
convertFreshnessToCron(catalogMaterializedTable.getDefinitionFreshness());
+// create full refresh job
+CreateRefreshWorkflow createRefreshWorkflow =
+new CreatePeriodicRefreshWorkflow(
+materializedTableIdentifier,
+catalogMaterializedTable.getDefinitionQuery(),
+cronExpression,
+catalogMaterializedTable.getOptions(),
+Collections.emptyMap(),
+restEndpointUrl);
+
+try {
+RefreshHandler refreshHandler =
+
workflowScheduler.createRefreshWorkflow(createRefreshWorkflow);
+RefreshHandlerSerializer refreshHandlerSerializer =
+workflowScheduler.getRefreshHandlerSerializer();
+byte[] serializedRefreshHandler =
refreshHandlerSerializer.serialize(refreshHandler);
+
+CatalogMaterializedTable updatedMaterializedTable =
Review Comment:
We can extract a util method as follows:
```
private void updateRefreshHandler(
OperationExecutor operationExecutor,
OperationHandle handle,
ObjectIdentifier materializedTableIdentifier,
CatalogMaterializedTable catalogMaterializedTable,
String refreshHandlerSummary,
byte[] refreshHandlerBytes) {
CatalogMaterializedTable updatedMaterializedTable =
catalogMaterializedTable.copy(
CatalogMaterializedTable.RefreshStatus.ACTIVATED,
refreshHandlerSummary,
refreshHandlerBytes);
List tableChanges = new ArrayList<>();
tableChanges.add(
TableChange.modifyRefreshStatus(CatalogMaterializedTable.RefreshStatus.ACTIVATED));
tableChanges.add(
TableChange.modifyRefreshHandler(refreshHandlerSummary,
refreshHandlerBytes));
AlterMaterializedTableChangeOperation
alterMaterializedTableChangeOperation =
new AlterMaterializedTableChangeOperation(
materializedTableIdentifier, tableChanges,
updatedMaterializedTable);
// update RefreshHandler to Catalog
operationExecutor.callExecutableOperation(handle,
alterMaterializedTableChangeOperation);
}
```
This method and `executeContinuousRefreshJob` can both reuse it.
##
flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManager.java:
##
@@ -165,7 +215,73 @@ private static void createMaterializedInContinuousMode(
}
}
-private static ResultFetcher callAlterMaterializedTableSuspend(
+private void
[jira] [Created] (FLINK-35493) Make max history age and count configurable for FlinkStateSnapshot resources
Mate Czagany created FLINK-35493: Summary: Make max history age and count configurable for FlinkStateSnapshot resources Key: FLINK-35493 URL: https://issues.apache.org/jira/browse/FLINK-35493 Project: Flink Issue Type: Sub-task Components: Kubernetes Operator Reporter: Mate Czagany -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35492) Add metrics for FlinkStateSnapshot resources
Mate Czagany created FLINK-35492: Summary: Add metrics for FlinkStateSnapshot resources Key: FLINK-35492 URL: https://issues.apache.org/jira/browse/FLINK-35492 Project: Flink Issue Type: Sub-task Components: Kubernetes Operator Reporter: Mate Czagany -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30719) flink-runtime-web failed due to a corrupted nodejs dependency
[
https://issues.apache.org/jira/browse/FLINK-30719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850700#comment-17850700
]
Weijie Guo commented on FLINK-30719:
{code:java}
The archive file
/__w/1/.m2/repository/com/github/eirslett/node/16.13.2/node-16.13.2-linux-x64.tar.gz
is corrupted and will be deleted. Please try the build again.
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59964&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb&l=9778
> flink-runtime-web failed due to a corrupted nodejs dependency
> -
>
> Key: FLINK-30719
> URL: https://issues.apache.org/jira/browse/FLINK-30719
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Web Frontend, Test Infrastructure, Tests
>Affects Versions: 1.16.0, 1.17.0, 1.18.0
>Reporter: Matthias Pohl
>Assignee: Sergey Nuyanzin
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44954&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb&l=12550
> The build failed due to a corrupted nodejs dependency:
> {code}
> [ERROR] The archive file
> /__w/1/.m2/repository/com/github/eirslett/node/16.13.2/node-16.13.2-linux-x64.tar.gz
> is corrupted and will be deleted. Please try the build again.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35489) Metaspace size can be too little after autotuning change memory setting
[
https://issues.apache.org/jira/browse/FLINK-35489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850694#comment-17850694
]
Nicolas Fraison edited comment on FLINK-35489 at 5/30/24 12:07 PM:
---
Thks [~fanrui] for the feedback it help me realise that my analysis was wrong.
The issue we are facing is the JVM crashing after the
[autotuning|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autotuning/]
change some memory config:
{code:java}
Starting kubernetes-taskmanager as a console application on host
flink-kafka-job-apache-right-taskmanager-1-1.
Exception in thread "main" *** java.lang.instrument ASSERTION FAILED ***:
"result" with message agent load/premain call failed at
src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart
failed
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted,
Vv=VM code, C=native code)
V [libjvm.so+0x78dee4] jni_FatalError+0x70
V [libjvm.so+0x88df00] JvmtiExport::post_vm_initialized()+0x240
V [libjvm.so+0xc353fc] Threads::create_vm(JavaVMInitArgs*, bool*)+0x7ac
V [libjvm.so+0x79c05c] JNI_CreateJavaVM+0x7c
C [libjli.so+0x3b2c] JavaMain+0x7c
C [libjli.so+0x7fdc] ThreadJavaMain+0xc
C [libpthread.so.0+0x7624] start_thread+0x184 {code}
Seeing this big increase of HEAP (from 1.5 to more than 3GB and the fact that
the memory.managed.size was shrink to 0b make me thing that it was linked to
missing off heap.
But you are right that jvm-overhead already reserved some memory for the off
heap (and we indeed have around 400 MB with that config)
So looking back to the new config I've identified the issue which is on the
jvm-metaspace having been shrink to 22MB while it was set at 256MB.
I've done a test increasing this parameter and the TM is now able to start.
For the meta space computation size I can see the autotuning computing
METASPACE_MEMORY_USED=1.41521584E8 which seems to be appropriate metaspace
sizing.
But due to the the memBudget management it ends up setting only 22MB to the
metaspace ([first allocate remaining memory to the heap and then this new
remaining to metaspace and finally to managed
memory|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/MemoryTuning.java#L130])
was (Author: JIRAUSER299678):
Thks [~fanrui] for the feedback it help me realise that my analysis was wrong.
The issue we are facing ifs the JVM crashing after the
[autotuning|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autotuning/]
change some memory config:
{code:java}
Starting kubernetes-taskmanager as a console application on host
flink-kafka-job-apache-right-taskmanager-1-1.
Exception in thread "main" *** java.lang.instrument ASSERTION FAILED ***:
"result" with message agent load/premain call failed at
src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart
failed
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted,
Vv=VM code, C=native code)
V [libjvm.so+0x78dee4] jni_FatalError+0x70
V [libjvm.so+0x88df00] JvmtiExport::post_vm_initialized()+0x240
V [libjvm.so+0xc353fc] Threads::create_vm(JavaVMInitArgs*, bool*)+0x7ac
V [libjvm.so+0x79c05c] JNI_CreateJavaVM+0x7c
C [libjli.so+0x3b2c] JavaMain+0x7c
C [libjli.so+0x7fdc] ThreadJavaMain+0xc
C [libpthread.so.0+0x7624] start_thread+0x184 {code}
Seeing this big increase of HEAP (from 1.5 to more than 3GB and the fact that
the memory.managed.size was shrink to 0b make me thing that it was linked to
missing off heap.
But you are right that jvm-overhead already reserved some memory for the off
heap (and we indeed have around 400 MB with that config)
So looking back to the new config I've identified the issue which is on the
jvm-metaspace having been shrink to 22MB while it was set at 256MB.
I've done a test increasing this parameter and the TM is now able to start.
For the meta space computation size I can see the autotuning computing
METASPACE_MEMORY_USED=1.41521584E8 which seems to be appropriate metaspace
sizing.
But due to the the memBudget management it ends up setting only 22MB to the
metaspace ([first allocate remaining memory to the heap and then this new
remaining to metaspace and finally to managed
memory|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/MemoryTuning.java#L130])
> Metaspace size can be too little after autotuning change memory setting
> ---
>
> Key: FLINK-35489
> URL: h
[jira] [Updated] (FLINK-35489) Metaspace size can be too little after autotuning change memory setting
[
https://issues.apache.org/jira/browse/FLINK-35489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Fraison updated FLINK-35489:
Description:
We have enable the autotuning feature on one of our flink job with below config
{code:java}
# Autoscaler configuration
job.autoscaler.enabled: "true"
job.autoscaler.stabilization.interval: 1m
job.autoscaler.metrics.window: 10m
job.autoscaler.target.utilization: "0.8"
job.autoscaler.target.utilization.boundary: "0.1"
job.autoscaler.restart.time: 2m
job.autoscaler.catch-up.duration: 10m
job.autoscaler.memory.tuning.enabled: true
job.autoscaler.memory.tuning.overhead: 0.5
job.autoscaler.memory.tuning.maximize-managed-memory: true{code}
During a scale down the autotuning decided to give all the memory to to JVM
(having heap being scale by 2) settting taskmanager.memory.managed.size to 0b.
Here is the config that was compute by the autotuning for a TM running on a 4GB
pod:
{code:java}
taskmanager.memory.network.max: 4063232b
taskmanager.memory.network.min: 4063232b
taskmanager.memory.jvm-overhead.max: 433791712b
taskmanager.memory.task.heap.size: 3699934605b
taskmanager.memory.framework.off-heap.size: 134217728b
taskmanager.memory.jvm-metaspace.size: 22960020b
taskmanager.memory.framework.heap.size: "0 bytes"
taskmanager.memory.flink.size: 3838215565b
taskmanager.memory.managed.size: 0b {code}
This has lead to some issue starting the TM because we are relying on some
javaagent performing some memory allocation outside of the JVM (rely on some C
bindings).
Tuning the overhead or disabling the scale-down-compensation.enabled could have
helped for that particular event but this can leads to other issue as it could
leads to too little HEAP size being computed.
It would be interesting to be able to set a min memory.managed.size to be taken
in account by the autotuning.
What do you think about this? Do you think that some other specific config
should have been applied to avoid this issue?
Edit see this comment that leads to the metaspace issue:
https://issues.apache.org/jira/browse/FLINK-35489?focusedCommentId=17850694&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17850694
was:
We have enable the autotuning feature on one of our flink job with below config
{code:java}
# Autoscaler configuration
job.autoscaler.enabled: "true"
job.autoscaler.stabilization.interval: 1m
job.autoscaler.metrics.window: 10m
job.autoscaler.target.utilization: "0.8"
job.autoscaler.target.utilization.boundary: "0.1"
job.autoscaler.restart.time: 2m
job.autoscaler.catch-up.duration: 10m
job.autoscaler.memory.tuning.enabled: true
job.autoscaler.memory.tuning.overhead: 0.5
job.autoscaler.memory.tuning.maximize-managed-memory: true{code}
During a scale down the autotuning decided to give all the memory to to JVM
(having heap being scale by 2) settting taskmanager.memory.managed.size to 0b.
Here is the config that was compute by the autotuning for a TM running on a 4GB
pod:
{code:java}
taskmanager.memory.network.max: 4063232b
taskmanager.memory.network.min: 4063232b
taskmanager.memory.jvm-overhead.max: 433791712b
taskmanager.memory.task.heap.size: 3699934605b
taskmanager.memory.framework.off-heap.size: 134217728b
taskmanager.memory.jvm-metaspace.size: 22960020b
taskmanager.memory.framework.heap.size: "0 bytes"
taskmanager.memory.flink.size: 3838215565b
taskmanager.memory.managed.size: 0b {code}
This has lead to some issue starting the TM because we are relying on some
javaagent performing some memory allocation outside of the JVM (rely on some C
bindings).
Tuning the overhead or disabling the scale-down-compensation.enabled could have
helped for that particular event but this can leads to other issue as it could
leads to too little HEAP size being computed.
It would be interesting to be able to set a min memory.managed.size to be taken
in account by the autotuning.
What do you think about this? Do you think that some other specific config
should have been applied to avoid this issue?
Edit
> Metaspace size can be too little after autotuning change memory setting
> ---
>
> Key: FLINK-35489
> URL: https://issues.apache.org/jira/browse/FLINK-35489
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.8.0
>Reporter: Nicolas Fraison
>Priority: Major
>
> We have enable the autotuning feature on one of our flink job with below
> config
> {code:java}
> # Autoscaler configuration
> job.autoscaler.enabled: "true"
> job.autoscaler.stabilization.interval: 1m
> job.autoscaler.metrics.window: 10m
> job.autoscaler.target.utilization: "0.8"
> job.autoscaler.target.utilization.boundary: "0.1"
> job.autoscaler.res
[jira] [Updated] (FLINK-35489) Metaspace size can be too little after autotuning change memory setting
[
https://issues.apache.org/jira/browse/FLINK-35489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Fraison updated FLINK-35489:
Description:
We have enable the autotuning feature on one of our flink job with below config
{code:java}
# Autoscaler configuration
job.autoscaler.enabled: "true"
job.autoscaler.stabilization.interval: 1m
job.autoscaler.metrics.window: 10m
job.autoscaler.target.utilization: "0.8"
job.autoscaler.target.utilization.boundary: "0.1"
job.autoscaler.restart.time: 2m
job.autoscaler.catch-up.duration: 10m
job.autoscaler.memory.tuning.enabled: true
job.autoscaler.memory.tuning.overhead: 0.5
job.autoscaler.memory.tuning.maximize-managed-memory: true{code}
During a scale down the autotuning decided to give all the memory to to JVM
(having heap being scale by 2) settting taskmanager.memory.managed.size to 0b.
Here is the config that was compute by the autotuning for a TM running on a 4GB
pod:
{code:java}
taskmanager.memory.network.max: 4063232b
taskmanager.memory.network.min: 4063232b
taskmanager.memory.jvm-overhead.max: 433791712b
taskmanager.memory.task.heap.size: 3699934605b
taskmanager.memory.framework.off-heap.size: 134217728b
taskmanager.memory.jvm-metaspace.size: 22960020b
taskmanager.memory.framework.heap.size: "0 bytes"
taskmanager.memory.flink.size: 3838215565b
taskmanager.memory.managed.size: 0b {code}
This has lead to some issue starting the TM because we are relying on some
javaagent performing some memory allocation outside of the JVM (rely on some C
bindings).
Tuning the overhead or disabling the scale-down-compensation.enabled could have
helped for that particular event but this can leads to other issue as it could
leads to too little HEAP size being computed.
It would be interesting to be able to set a min memory.managed.size to be taken
in account by the autotuning.
What do you think about this? Do you think that some other specific config
should have been applied to avoid this issue?
Edit
was:
We have enable the autotuning feature on one of our flink job with below config
{code:java}
# Autoscaler configuration
job.autoscaler.enabled: "true"
job.autoscaler.stabilization.interval: 1m
job.autoscaler.metrics.window: 10m
job.autoscaler.target.utilization: "0.8"
job.autoscaler.target.utilization.boundary: "0.1"
job.autoscaler.restart.time: 2m
job.autoscaler.catch-up.duration: 10m
job.autoscaler.memory.tuning.enabled: true
job.autoscaler.memory.tuning.overhead: 0.5
job.autoscaler.memory.tuning.maximize-managed-memory: true{code}
During a scale down the autotuning decided to give all the memory to to JVM
(having heap being scale by 2) settting taskmanager.memory.managed.size to 0b.
Here is the config that was compute by the autotuning for a TM running on a 4GB
pod:
{code:java}
taskmanager.memory.network.max: 4063232b
taskmanager.memory.network.min: 4063232b
taskmanager.memory.jvm-overhead.max: 433791712b
taskmanager.memory.task.heap.size: 3699934605b
taskmanager.memory.framework.off-heap.size: 134217728b
taskmanager.memory.jvm-metaspace.size: 22960020b
taskmanager.memory.framework.heap.size: "0 bytes"
taskmanager.memory.flink.size: 3838215565b
taskmanager.memory.managed.size: 0b {code}
This has lead to some issue starting the TM because we are relying on some
javaagent performing some memory allocation outside of the JVM (rely on some C
bindings).
Tuning the overhead or disabling the scale-down-compensation.enabled could have
helped for that particular event but this can leads to other issue as it could
leads to too little HEAP size being computed.
It would be interesting to be able to set a min memory.managed.size to be taken
in account by the autotuning.
What do you think about this? Do you think that some other specific config
should have been applied to avoid this issue?
> Metaspace size can be too little after autotuning change memory setting
> ---
>
> Key: FLINK-35489
> URL: https://issues.apache.org/jira/browse/FLINK-35489
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.8.0
>Reporter: Nicolas Fraison
>Priority: Major
>
> We have enable the autotuning feature on one of our flink job with below
> config
> {code:java}
> # Autoscaler configuration
> job.autoscaler.enabled: "true"
> job.autoscaler.stabilization.interval: 1m
> job.autoscaler.metrics.window: 10m
> job.autoscaler.target.utilization: "0.8"
> job.autoscaler.target.utilization.boundary: "0.1"
> job.autoscaler.restart.time: 2m
> job.autoscaler.catch-up.duration: 10m
> job.autoscaler.memory.tuning.enabled: true
> job.autoscaler.memory.tuning.overhead: 0.5
> job.autoscaler.memory.tuning.maximize-managed-memory: true{code}
> During a scale
[jira] [Updated] (FLINK-35489) Metaspace size can be too little after autotuning change memory setting
[
https://issues.apache.org/jira/browse/FLINK-35489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Fraison updated FLINK-35489:
Summary: Metaspace size can be too little after autotuning change memory
setting (was: Add capability to set min taskmanager.memory.managed.size when
enabling autotuning)
> Metaspace size can be too little after autotuning change memory setting
> ---
>
> Key: FLINK-35489
> URL: https://issues.apache.org/jira/browse/FLINK-35489
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.8.0
>Reporter: Nicolas Fraison
>Priority: Major
>
> We have enable the autotuning feature on one of our flink job with below
> config
> {code:java}
> # Autoscaler configuration
> job.autoscaler.enabled: "true"
> job.autoscaler.stabilization.interval: 1m
> job.autoscaler.metrics.window: 10m
> job.autoscaler.target.utilization: "0.8"
> job.autoscaler.target.utilization.boundary: "0.1"
> job.autoscaler.restart.time: 2m
> job.autoscaler.catch-up.duration: 10m
> job.autoscaler.memory.tuning.enabled: true
> job.autoscaler.memory.tuning.overhead: 0.5
> job.autoscaler.memory.tuning.maximize-managed-memory: true{code}
> During a scale down the autotuning decided to give all the memory to to JVM
> (having heap being scale by 2) settting taskmanager.memory.managed.size to 0b.
> Here is the config that was compute by the autotuning for a TM running on a
> 4GB pod:
> {code:java}
> taskmanager.memory.network.max: 4063232b
> taskmanager.memory.network.min: 4063232b
> taskmanager.memory.jvm-overhead.max: 433791712b
> taskmanager.memory.task.heap.size: 3699934605b
> taskmanager.memory.framework.off-heap.size: 134217728b
> taskmanager.memory.jvm-metaspace.size: 22960020b
> taskmanager.memory.framework.heap.size: "0 bytes"
> taskmanager.memory.flink.size: 3838215565b
> taskmanager.memory.managed.size: 0b {code}
> This has lead to some issue starting the TM because we are relying on some
> javaagent performing some memory allocation outside of the JVM (rely on some
> C bindings).
> Tuning the overhead or disabling the scale-down-compensation.enabled could
> have helped for that particular event but this can leads to other issue as it
> could leads to too little HEAP size being computed.
> It would be interesting to be able to set a min memory.managed.size to be
> taken in account by the autotuning.
> What do you think about this? Do you think that some other specific config
> should have been applied to avoid this issue?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35406] Use inner serializer when casting RAW type to BINARY or… [flink]
twalthr commented on code in PR #24818:
URL: https://github.com/apache/flink/pull/24818#discussion_r1620578309
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RawToBinaryCastRule.java:
##
@@ -62,7 +62,7 @@ public boolean canFail(LogicalType inputLogicalType,
LogicalType targetLogicalTy
// new behavior
isNull$290 = isNull$289;
if (!isNull$290) {
-byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292);
+byte[] deserializedByteArray$76 =
result$289.toBytes(typeSerializer$292.getInnerSerializer());
Review Comment:
Why do we actually need the call to `getInnerSerializer()` during runtime
shouldn't we simply use this serializer as the `typeSerializer$292` for code
generation? Or does the implementation not allow this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-35489) Add capability to set min taskmanager.memory.managed.size when enabling autotuning
[
https://issues.apache.org/jira/browse/FLINK-35489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850694#comment-17850694
]
Nicolas Fraison commented on FLINK-35489:
-
Thks [~fanrui] for the feedback it help me realise that my analysis was wrong.
The issue we are facing ifs the JVM crashing after the
[autotuning|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autotuning/]
change some memory config:
{code:java}
Starting kubernetes-taskmanager as a console application on host
flink-kafka-job-apache-right-taskmanager-1-1.
Exception in thread "main" *** java.lang.instrument ASSERTION FAILED ***:
"result" with message agent load/premain call failed at
src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart
failed
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted,
Vv=VM code, C=native code)
V [libjvm.so+0x78dee4] jni_FatalError+0x70
V [libjvm.so+0x88df00] JvmtiExport::post_vm_initialized()+0x240
V [libjvm.so+0xc353fc] Threads::create_vm(JavaVMInitArgs*, bool*)+0x7ac
V [libjvm.so+0x79c05c] JNI_CreateJavaVM+0x7c
C [libjli.so+0x3b2c] JavaMain+0x7c
C [libjli.so+0x7fdc] ThreadJavaMain+0xc
C [libpthread.so.0+0x7624] start_thread+0x184 {code}
Seeing this big increase of HEAP (from 1.5 to more than 3GB and the fact that
the memory.managed.size was shrink to 0b make me thing that it was linked to
missing off heap.
But you are right that jvm-overhead already reserved some memory for the off
heap (and we indeed have around 400 MB with that config)
So looking back to the new config I've identified the issue which is on the
jvm-metaspace having been shrink to 22MB while it was set at 256MB.
I've done a test increasing this parameter and the TM is now able to start.
For the meta space computation size I can see the autotuning computing
METASPACE_MEMORY_USED=1.41521584E8 which seems to be appropriate metaspace
sizing.
But due to the the memBudget management it ends up setting only 22MB to the
metaspace ([first allocate remaining memory to the heap and then this new
remaining to metaspace and finally to managed
memory|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/MemoryTuning.java#L130])
> Add capability to set min taskmanager.memory.managed.size when enabling
> autotuning
> --
>
> Key: FLINK-35489
> URL: https://issues.apache.org/jira/browse/FLINK-35489
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.8.0
>Reporter: Nicolas Fraison
>Priority: Major
>
> We have enable the autotuning feature on one of our flink job with below
> config
> {code:java}
> # Autoscaler configuration
> job.autoscaler.enabled: "true"
> job.autoscaler.stabilization.interval: 1m
> job.autoscaler.metrics.window: 10m
> job.autoscaler.target.utilization: "0.8"
> job.autoscaler.target.utilization.boundary: "0.1"
> job.autoscaler.restart.time: 2m
> job.autoscaler.catch-up.duration: 10m
> job.autoscaler.memory.tuning.enabled: true
> job.autoscaler.memory.tuning.overhead: 0.5
> job.autoscaler.memory.tuning.maximize-managed-memory: true{code}
> During a scale down the autotuning decided to give all the memory to to JVM
> (having heap being scale by 2) settting taskmanager.memory.managed.size to 0b.
> Here is the config that was compute by the autotuning for a TM running on a
> 4GB pod:
> {code:java}
> taskmanager.memory.network.max: 4063232b
> taskmanager.memory.network.min: 4063232b
> taskmanager.memory.jvm-overhead.max: 433791712b
> taskmanager.memory.task.heap.size: 3699934605b
> taskmanager.memory.framework.off-heap.size: 134217728b
> taskmanager.memory.jvm-metaspace.size: 22960020b
> taskmanager.memory.framework.heap.size: "0 bytes"
> taskmanager.memory.flink.size: 3838215565b
> taskmanager.memory.managed.size: 0b {code}
> This has lead to some issue starting the TM because we are relying on some
> javaagent performing some memory allocation outside of the JVM (rely on some
> C bindings).
> Tuning the overhead or disabling the scale-down-compensation.enabled could
> have helped for that particular event but this can leads to other issue as it
> could leads to too little HEAP size being computed.
> It would be interesting to be able to set a min memory.managed.size to be
> taken in account by the autotuning.
> What do you think about this? Do you think that some other specific config
> should have been applied to avoid this issue?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [docs] Improve the readme and issue template [flink-cdc]
caicancai commented on code in PR #3383: URL: https://github.com/apache/flink-cdc/pull/3383#discussion_r1620475491 ## .github/ISSUE_TEMPLATE/issue-notice.yml: ## @@ -47,6 +50,8 @@ body: 感谢您选择 Flink CDC! 由于 Apache Flink 的要求，请在 [Apache Jira](https://issues.apache.org/jira) 的 `Flink` 项目下使用 `Flink CDC` 标签来反馈缺陷或新功能。 +您需要登录 Jira 帐户才能创建 issue 去记录缺陷或新功能，如果您没有 Apache JIRA 帐户，您可以在 [ASF Jira 账号](https://selfserve.apache.org/jira-account.html) +申请账号，账号创建会由申请项目的 PMC member 审核，通常一到两个工作日可以通过。 Review Comment: ASF Self-serve portal == ASF Jira 账号？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33001) KafkaSource in batch mode failing with exception if topic partition is empty
[
https://issues.apache.org/jira/browse/FLINK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850678#comment-17850678
]
Naci Simsek commented on FLINK-33001:
-
Just be able to try with the *flink-connector-kafka 3.1.0-1.18* on *Flink
1.18.1* session cluster with a batch job, and the exception is not there
anymore. The job finishes gracefully, without setting any specific log level
for any subclasses. I have the following log setting:
{code:java}
rootLogger.level = TRACE {code}
Therefore, the issue seems to be fixed.
However, there is another issue when setting a specific offset to start reading
from, which causes an offset reset and endless loop of
FetchTasks, so that the batch job keeps running, till there is an event
received into the topic, but of course this should be a content of another
ticket.
> KafkaSource in batch mode failing with exception if topic partition is empty
>
>
> Key: FLINK-33001
> URL: https://issues.apache.org/jira/browse/FLINK-33001
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.7, 1.14.6, 1.17.1
>Reporter: Abdul
>Priority: Major
>
> If the Kafka topic is empty in Batch mode, there is an exception while
> processing it. This bug was supposedly fixed but unfortunately, the exception
> still occurs. The original bug was reported as this
> https://issues.apache.org/jira/browse/FLINK-27041
> We tried to backport it but it still doesn't work.
> * The problem will occur in case of the DEBUG level of logger for class
> org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader
> * The same problems will occur in other versions of Flink, at least in the
> 1.15 release branch and tag release-1.15.4
> * The same problem also occurs in Flink 1.17.1 and 1.14
>
> The minimal code to produce this is
>
> {code:java}
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(RuntimeExecutionMode.BATCH);
> KafkaSource kafkaSource = KafkaSource
> .builder()
> .setBootstrapServers("localhost:9092")
> .setTopics("test_topic")
> .setValueOnlyDeserializer(new
> SimpleStringSchema())
> .setBounded(OffsetsInitializer.latest())
> .build();
> DataStream stream = env.fromSource(
> kafkaSource,
> WatermarkStrategy.noWatermarks(),
> "Kafka Source"
> );
> stream.print();
> env.execute("Flink KafkaSource test job"); {code}
> This produces exception:
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:199)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:154)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:116)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:275)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:67)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:398)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:583)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573) at
> java.lang.Thread.run(Thread.java:748)Caused by: java.lang.RuntimeException:
> SplitFetcher thread 0 received unexpected exception while polling the records
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:146)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:26
[jira] [Comment Edited] (FLINK-33001) KafkaSource in batch mode failing with exception if topic partition is empty
[
https://issues.apache.org/jira/browse/FLINK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850678#comment-17850678
]
Naci Simsek edited comment on FLINK-33001 at 5/30/24 10:41 AM:
---
Just be able to try with the *flink-connector-kafka 3.1.0-1.18* on *Flink
1.18.1* session cluster with a batch job, and the exception is not there
anymore. The job finishes gracefully, without setting any specific log level
for any subclasses. I have the following log setting:
{code:java}
rootLogger.level = TRACE {code}
Therefore, the issue seems to be fixed.
However, there is another issue when setting a specific offset to start reading
from, which causes an offset reset and endless loop of FetchTasks, so that the
batch job keeps running, till there is an event received into the topic, but of
course this should be a content of another ticket.
was (Author: JIRAUSER302646):
Just be able to try with the *flink-connector-kafka 3.1.0-1.18* on *Flink
1.18.1* session cluster with a batch job, and the exception is not there
anymore. The job finishes gracefully, without setting any specific log level
for any subclasses. I have the following log setting:
{code:java}
rootLogger.level = TRACE {code}
Therefore, the issue seems to be fixed.
However, there is another issue when setting a specific offset to start reading
from, which causes an offset reset and endless loop of
FetchTasks, so that the batch job keeps running, till there is an event
received into the topic, but of course this should be a content of another
ticket.
> KafkaSource in batch mode failing with exception if topic partition is empty
>
>
> Key: FLINK-33001
> URL: https://issues.apache.org/jira/browse/FLINK-33001
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.7, 1.14.6, 1.17.1
>Reporter: Abdul
>Priority: Major
>
> If the Kafka topic is empty in Batch mode, there is an exception while
> processing it. This bug was supposedly fixed but unfortunately, the exception
> still occurs. The original bug was reported as this
> https://issues.apache.org/jira/browse/FLINK-27041
> We tried to backport it but it still doesn't work.
> * The problem will occur in case of the DEBUG level of logger for class
> org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader
> * The same problems will occur in other versions of Flink, at least in the
> 1.15 release branch and tag release-1.15.4
> * The same problem also occurs in Flink 1.17.1 and 1.14
>
> The minimal code to produce this is
>
> {code:java}
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(RuntimeExecutionMode.BATCH);
> KafkaSource kafkaSource = KafkaSource
> .builder()
> .setBootstrapServers("localhost:9092")
> .setTopics("test_topic")
> .setValueOnlyDeserializer(new
> SimpleStringSchema())
> .setBounded(OffsetsInitializer.latest())
> .build();
> DataStream stream = env.fromSource(
> kafkaSource,
> WatermarkStrategy.noWatermarks(),
> "Kafka Source"
> );
> stream.print();
> env.execute("Flink KafkaSource test job"); {code}
> This produces exception:
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:199)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:154)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:116)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:275)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:67)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:398)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619)
> at
> org.apache.flink.streaming.runtime.tasks
[jira] [Comment Edited] (FLINK-33001) KafkaSource in batch mode failing with exception if topic partition is empty
[
https://issues.apache.org/jira/browse/FLINK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850678#comment-17850678
]
Naci Simsek edited comment on FLINK-33001 at 5/30/24 10:40 AM:
---
Just be able to try with the *flink-connector-kafka 3.1.0-1.18* on *Flink
1.18.1* session cluster with a batch job, and the exception is not there
anymore. The job finishes gracefully, without setting any specific log level
for any subclasses. I have the following log setting:
{code:java}
rootLogger.level = TRACE {code}
Therefore, the issue seems to be fixed.
However, there is another issue when setting a specific offset to start reading
from, which causes an offset reset and endless loop of
FetchTasks, so that the batch job keeps running, till there is an event
received into the topic, but of course this should be a content of another
ticket.
was (Author: JIRAUSER302646):
Just be able to try with the *flink-connector-kafka 3.1.0-1.18* on *Flink
1.18.1* session cluster with a batch job, and the exception is not there
anymore. The job finishes gracefully, without setting any specific log level
for any subclasses. I have the following log setting:
{code:java}
rootLogger.level = TRACE {code}
Therefore, the issue seems to be fixed.
However, there is another issue when setting a specific offset to start reading
from, which causes an offset reset and endless loop of
FetchTasks, so that the batch job keeps running, till there is an event
received into the topic, but of course this should be a content of another
ticket.
> KafkaSource in batch mode failing with exception if topic partition is empty
>
>
> Key: FLINK-33001
> URL: https://issues.apache.org/jira/browse/FLINK-33001
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.7, 1.14.6, 1.17.1
>Reporter: Abdul
>Priority: Major
>
> If the Kafka topic is empty in Batch mode, there is an exception while
> processing it. This bug was supposedly fixed but unfortunately, the exception
> still occurs. The original bug was reported as this
> https://issues.apache.org/jira/browse/FLINK-27041
> We tried to backport it but it still doesn't work.
> * The problem will occur in case of the DEBUG level of logger for class
> org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader
> * The same problems will occur in other versions of Flink, at least in the
> 1.15 release branch and tag release-1.15.4
> * The same problem also occurs in Flink 1.17.1 and 1.14
>
> The minimal code to produce this is
>
> {code:java}
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(RuntimeExecutionMode.BATCH);
> KafkaSource kafkaSource = KafkaSource
> .builder()
> .setBootstrapServers("localhost:9092")
> .setTopics("test_topic")
> .setValueOnlyDeserializer(new
> SimpleStringSchema())
> .setBounded(OffsetsInitializer.latest())
> .build();
> DataStream stream = env.fromSource(
> kafkaSource,
> WatermarkStrategy.noWatermarks(),
> "Kafka Source"
> );
> stream.print();
> env.execute("Flink KafkaSource test job"); {code}
> This produces exception:
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:199)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:154)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:116)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:275)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:67)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:398)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619)
> at
> org.apache.flink.streaming.runtime.t
Re: [PR] [FLINK-34239][core] Add copy() method for SerializerConfig [flink]
kumar-mallikarjuna commented on PR #24544: URL: https://github.com/apache/flink/pull/24544#issuecomment-2139268835 Done, @reswqa . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-35484) Flink document file had removed but website can access
[ https://issues.apache.org/jira/browse/FLINK-35484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu resolved FLINK-35484. Resolution: Fixed Fixed via master: b1841c0ff34b6e8d118139f6f87b7fdf06afacd3 > Flink document file had removed but website can access > -- > > Key: FLINK-35484 > URL: https://issues.apache.org/jira/browse/FLINK-35484 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Fix For: 1.20.0 > > > Flink 1.18 document had remove document about DataSet : issue link > https://issues.apache.org/jira/browse/FLINK-32741. > But I can still access the link : > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35484][doc-ci] Flink document file had removed but website can access [flink]
leonardBang merged PR #24870: URL: https://github.com/apache/flink/pull/24870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35363] FLIP-449: Reorganization of flink-connector-jdbc [flink-connector-jdbc]
eskabetxe commented on code in PR #123:
URL:
https://github.com/apache/flink-connector-jdbc/pull/123#discussion_r1620403861
##
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/core/table/JdbcFactory.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.core.table;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.jdbc.core.table.catalog.JdbcCatalog;
+import org.apache.flink.connector.jdbc.core.table.dialect.JdbcDialect;
+import org.apache.flink.util.StringUtils;
+
+/**
+ * A factory to create a specific {@link JdbcDialect}. This factory is used
with Java's Service
+ * Provider Interfaces (SPI) for discovering.
+ *
+ * Classes that implement this interface can be added to the
+ * "META_INF/services/org.apache.flink.connector.jdbc.core.table.JdbcFactory"
file of a JAR file in
+ * the current classpath to be found.
+ *
+ * @see JdbcDialect
+ */
+@PublicEvolving
+public interface JdbcFactory {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35463]Fixed issue for route rule changed when restored from checkpoint. [flink-cdc]
hk-lrzy commented on PR #3364: URL: https://github.com/apache/flink-cdc/pull/3364#issuecomment-2139205095 > exception got it, let me do the double check and give you reply asap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source [flink-connector-jdbc]
1996fanrui commented on code in PR #119: URL: https://github.com/apache/flink-connector-jdbc/pull/119#discussion_r1620372109 ## flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/utils/ContinuousUnBoundingSettings.java: ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.connector.jdbc.utils; + +import org.apache.flink.annotation.PublicEvolving; + +import java.io.Serializable; +import java.time.Duration; +import java.util.Objects; + +import static org.apache.flink.util.Preconditions.checkNotNull; + +/** + * Settings describing how to do continuous file discovery and enumeration for the file source's + * continuous discovery and streaming mode. + */ +@PublicEvolving Review Comment: ```suggestion @Experimental ``` We should use `Experimental` for the first version, and update it to `PublicEvolving` after it's stable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35491) [JUnit5 Migration] Module: Flink CDC modules
Muhammet Orazov created FLINK-35491: --- Summary: [JUnit5 Migration] Module: Flink CDC modules Key: FLINK-35491 URL: https://issues.apache.org/jira/browse/FLINK-35491 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Muhammet Orazov Migrate Junit4 tests to Junit5 for the following modules: * flink-cdc-common * flink-cdc-composer * flink-cdc-runtime * flink-cdc-connect/flink-cdc-pipeline-connectors * flink-cdc-e2e-tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32384) Remove deprecated configuration keys which violate YAML spec
[ https://issues.apache.org/jira/browse/FLINK-32384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850664#comment-17850664 ] Kartikey Pant commented on FLINK-32384: --- Hello [~zhuzh], I am eager to start contributing to the Apache Flink community. May I take on this issue to work on? > Remove deprecated configuration keys which violate YAML spec > > > Key: FLINK-32384 > URL: https://issues.apache.org/jira/browse/FLINK-32384 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Reporter: Zhu Zhu >Priority: Major > Labels: 2.0-related > Fix For: 2.0.0 > > > In FLINK-29372, key that violate YAML spec are renamed to a valid form and > the old names are deprecated. > In Flink 2.0 we should remove these deprecated keys. This will prevent users > (unintentionally) to create invalid YAML form flink-conf.yaml. > Then with the work of FLINK-23620, we can remove the non-standard YAML > parsing logic and enforce standard YAML validation in CI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35490) [JUnit5 Migration] Module: Flink CDC flink-cdc-connect/flink-cdc-source-connectors
[ https://issues.apache.org/jira/browse/FLINK-35490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850663#comment-17850663 ] Muhammet Orazov commented on FLINK-35490: - I would be happy to work on this, please assign it to me, thanks (y) > [JUnit5 Migration] Module: Flink CDC > flink-cdc-connect/flink-cdc-source-connectors > -- > > Key: FLINK-35490 > URL: https://issues.apache.org/jira/browse/FLINK-35490 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Muhammet Orazov >Priority: Major > > Migrate Junit4 tests to Junit5 in the Flink CDC following modules: > > - flink-cdc-connect/flink-cdc-source-connectors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35490) [JUnit5 Migration] Module: Flink CDC flink-cdc-connect/flink-cdc-source-connectors
Muhammet Orazov created FLINK-35490: --- Summary: [JUnit5 Migration] Module: Flink CDC flink-cdc-connect/flink-cdc-source-connectors Key: FLINK-35490 URL: https://issues.apache.org/jira/browse/FLINK-35490 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Muhammet Orazov Migrate Junit4 tests to Junit5 in the Flink CDC following modules: - flink-cdc-connect/flink-cdc-source-connectors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34585) [JUnit5 Migration] Module: Flink CDC
[ https://issues.apache.org/jira/browse/FLINK-34585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850662#comment-17850662 ] Muhammet Orazov commented on FLINK-34585: - Hey all, Since this is lots of effort, I suggest we divide this ticket into two parts. With the following modules for each ticket: First ticket to include the following module: * flink-cdc-connect/flink-cdc-source-connectors And the second ticket to include the following modules: * flink-cdc-common * flink-cdc-composer * flink-cdc-runtime * flink-cdc-connect/flink-cdc-pipeline-connectors * flink-cdc-e2e-tests Because the `flink-cdc-source-connectors` is big part that contains the Junit4 tests. > [JUnit5 Migration] Module: Flink CDC > > > Key: FLINK-34585 > URL: https://issues.apache.org/jira/browse/FLINK-34585 > Project: Flink > Issue Type: Sub-task > Components: Flink CDC >Reporter: Hang Ruan >Assignee: LvYanquan >Priority: Major > > Most tests in Flink CDC are still using Junit 4. We need to use Junit 5 > instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35484][doc-ci] Flink document file had removed but website can access [flink]
leonardBang commented on PR #24870: URL: https://github.com/apache/flink/pull/24870#issuecomment-2139125612 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35463]Fixed issue for route rule changed when restored from checkpoint. [flink-cdc]
yuxiqian commented on PR #3364: URL: https://github.com/apache/flink-cdc/pull/3364#issuecomment-2139108465 Hi @hk-lrzy, I've compiled Flink CDC on your branch and tried to verify this patch by doing the following steps: 1. Create a MySQL source with tables (`alice`, `bob`, and `chris`) in `fallen` DB: 2. Submit a pipeline job defined with the following YAML: ```yaml source: type: mysql tables: fallen.\.* # ... sink: type: values ``` 3. Send 4 data records with tag 1 to 4, waiting it to be recieved by sink. 4. Stop the job and create a savepoint file with `./bin/flink stop`. 5. Create three more tables (`dorothy`, `eve`, and `faye`) in the same DB, and insert data records with tag 5 to 8. 6. Resubmit the pipeline job again (which should capture all 6 tables now). --- Expected result: pipeline job successfully captured newly added tables. Actual result: pipeline job failed with the following exception: ``` org.apache.flink.shaded.guava31.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Schema is never registered or outdated for table "fallen.eve" at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2055) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache.get(LocalCache.java:3966) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950) at org.apache.flink.cdc.runtime.partitioning.PrePartitionOperator.partitionBy(PrePartitionOperator.java:102) at org.apache.flink.cdc.runtime.partitioning.PrePartitionOperator.processElement(PrePartitionOperator.java:92) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) at org.apache.flink.cdc.runtime.operators.schema.SchemaOperator.processElement(SchemaOperator.java:187) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:309) at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110) at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter$OutputCollector.collect(MySqlRecordEmitter.java:147) at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4856) at org.apache.flink.cdc.debezium.event.DebeziumEventDeserializationSchema.deserialize(DebeziumEventDeserializationSchema.java:93) at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitElement(MySqlRecordEmitter.java:120) at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.processElement(MySqlRecordEmitter.java:101) at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlPipelineRecordEmitter.processElement(MySqlPipelineRecordEmitter.java:120) at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:73) at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:46) at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:160) at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419) at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) at org.apache.flink.runtime.taskmanager.Ta
Re: [PR] [FLINK-35363] FLIP-449: Reorganization of flink-connector-jdbc [flink-connector-jdbc]
eskabetxe commented on code in PR #123:
URL:
https://github.com/apache/flink-connector-jdbc/pull/123#discussion_r1620277111
##
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/core/table/JdbcFactory.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.core.table;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.jdbc.core.table.catalog.JdbcCatalog;
+import org.apache.flink.connector.jdbc.core.table.dialect.JdbcDialect;
+import org.apache.flink.util.StringUtils;
+
+/**
+ * A factory to create a specific {@link JdbcDialect}. This factory is used
with Java's Service
+ * Provider Interfaces (SPI) for discovering.
+ *
+ * Classes that implement this interface can be added to the
+ * "META_INF/services/org.apache.flink.connector.jdbc.core.table.JdbcFactory"
file of a JAR file in
+ * the current classpath to be found.
+ *
+ * @see JdbcDialect
+ */
+@PublicEvolving
+public interface JdbcFactory {
Review Comment:
Thanks for you review @RocMarshal
point 1: I will change it as we talk offline..
point 2: Fixed, was a dependency problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35068][core][type] Introduce built-in serialization support for java.util.Set [flink]
jeyhunkarimov commented on code in PR #24845:
URL: https://github.com/apache/flink/pull/24845#discussion_r1620282539
##
flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/SetSerializer.java:
##
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeutils.base;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A serializer for {@link java.util.Set}. The serializer relies on an element
serializer for the
+ * serialization of the set's elements.
+ *
+ * The serialization format for the set is as follows: four bytes for the
length of the set,
+ * followed by the serialized representation of each element.
+ *
+ * @param The type of element in the set.
+ */
+@Internal
+public final class SetSerializer extends TypeSerializer> {
+
+private static final long serialVersionUID = 1L;
+
+/** The serializer for the elements of the set. */
+private final TypeSerializer elementSerializer;
+
+/**
+ * Creates a set serializer that uses the given serializer to serialize
the set's elements.
+ *
+ * @param elementSerializer The serializer for the elements of the set
+ */
+public SetSerializer(TypeSerializer elementSerializer) {
+this.elementSerializer = checkNotNull(elementSerializer);
+}
+
+//
+// SetSerializer specific properties
+//
+
+/**
+ * Gets the serializer for the elements of the set.
+ *
+ * @return The serializer for the elements of the set
+ */
+public TypeSerializer getElementSerializer() {
+return elementSerializer;
+}
+
+//
+// Type Serializer implementation
+//
+
+@Override
+public boolean isImmutableType() {
+return false;
+}
+
+@Override
+public TypeSerializer> duplicate() {
+TypeSerializer duplicateElement = elementSerializer.duplicate();
+return duplicateElement == elementSerializer ? this : new
SetSerializer<>(duplicateElement);
+}
+
+@Override
+public Set createInstance() {
+return new HashSet<>(0);
+}
+
+@Override
+public Set copy(Set from) {
+Set newSet = new HashSet<>(from.size());
+for (T element : from) {
+newSet.add(elementSerializer.copy(element));
+}
+return newSet;
+}
+
+@Override
+public Set copy(Set from, Set reuse) {
+return copy(from);
+}
+
+@Override
+public int getLength() {
+return -1; // var length
+}
+
+@Override
+public void serialize(Set set, DataOutputView target) throws
IOException {
+final int size = set.size();
+target.writeInt(size);
+for (T element : set) {
+elementSerializer.serialize(element, target);
+}
+}
+
+@Override
+public Set deserialize(DataInputView source) throws IOException {
+final int size = source.readInt();
+final Set set = new HashSet<>(size);
Review Comment:
Makes sense. Thanks for the clarification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35363] FLIP-449: Reorganization of flink-connector-jdbc [flink-connector-jdbc]
eskabetxe commented on code in PR #123:
URL:
https://github.com/apache/flink-connector-jdbc/pull/123#discussion_r1620277111
##
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/core/table/JdbcFactory.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.core.table;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.jdbc.core.table.catalog.JdbcCatalog;
+import org.apache.flink.connector.jdbc.core.table.dialect.JdbcDialect;
+import org.apache.flink.util.StringUtils;
+
+/**
+ * A factory to create a specific {@link JdbcDialect}. This factory is used
with Java's Service
+ * Provider Interfaces (SPI) for discovering.
+ *
+ * Classes that implement this interface can be added to the
+ * "META_INF/services/org.apache.flink.connector.jdbc.core.table.JdbcFactory"
file of a JAR file in
+ * the current classpath to be found.
+ *
+ * @see JdbcDialect
+ */
+@PublicEvolving
+public interface JdbcFactory {
Review Comment:
Thanks for you review @RocMarshal
point 1: I will change it as we talk offline..
point 2: Fixed, was a dependency problem//
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35363] FLIP-449: Reorganization of flink-connector-jdbc [flink-connector-jdbc]
eskabetxe commented on code in PR #123:
URL:
https://github.com/apache/flink-connector-jdbc/pull/123#discussion_r1620277111
##
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/core/table/JdbcFactory.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.core.table;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.jdbc.core.table.catalog.JdbcCatalog;
+import org.apache.flink.connector.jdbc.core.table.dialect.JdbcDialect;
+import org.apache.flink.util.StringUtils;
+
+/**
+ * A factory to create a specific {@link JdbcDialect}. This factory is used
with Java's Service
+ * Provider Interfaces (SPI) for discovering.
+ *
+ * Classes that implement this interface can be added to the
+ * "META_INF/services/org.apache.flink.connector.jdbc.core.table.JdbcFactory"
file of a JAR file in
+ * the current classpath to be found.
+ *
+ * @see JdbcDialect
+ */
+@PublicEvolving
+public interface JdbcFactory {
Review Comment:
point 1: I will change it as we talk offline..
point 2: Fixed, was a dependency problem//
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33937) Translate "Stateful Stream Processing" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-33937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850604#comment-17850604 ] Pinrui Dong edited comment on FLINK-33937 at 5/30/24 8:40 AM: -- It is still unresolved. Could you please assign it to me? [~jark] was (Author: JIRAUSER305528): It is still unsolved. Could you please assign it to me? [~jark] > Translate "Stateful Stream Processing" page into Chinese > > > Key: FLINK-33937 > URL: https://issues.apache.org/jira/browse/FLINK-33937 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Affects Versions: 1.8.0 >Reporter: Yongping Li >Priority: Major > Labels: translate > Attachments: image-2023-12-26-08-54-14-041.png > > > The page is located at > _"docs/content.zh/docs/concepts/stateful-stream-processing.md"_ > _!image-2023-12-26-08-54-14-041.png!_ -- This message was sent by Atlassian Jira (v8.20.10#820010)
