Re: [PR] [FLINK-35119][cdc-runtime] Change DataChangeEvent serialization and eserialization [flink-cdc]
zhongqishang commented on code in PR #3226:
URL: https://github.com/apache/flink-cdc/pull/3226#discussion_r1581741205
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/event/DataChangeEventSerializer.java:
##
@@ -76,28 +72,18 @@ public DataChangeEvent deserialize(DataInputView source)
throws IOException {
OperationType op = opSerializer.deserialize(source);
TableId tableId = tableIdSerializer.deserialize(source);
+RecordData before = recordDataSerializer.deserialize(source);
+RecordData after = recordDataSerializer.deserialize(source);
+Map meta = metaSerializer.deserialize(source);
Review Comment:
@yuxiqian Thanks for your review.
> Is it safe to deserialize here if `op` isn't any known following one?
IMO, the currently deserialized fields and order are consistent with those
during serialization. Even if an unknown op operation occurs, deserialization
will not cause an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Fix during the snapshot phase, projection is not being performed according to the user defined schema. [flink-cdc]
hzjhjjyy commented on PR #2495: URL: https://github.com/apache/flink-cdc/pull/2495#issuecomment-2080353903 > Hi @hzjhjjyy, thanks for your contribution! Could you please rebase your PR with latest `master` branch before it could be merged? You may need to rename packages like from `com.ververica.cdc` to `org.apache.flink.cdc`. Yes, I'll do it within the next couple of days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35248) Support table/column comment to modify DDL
melin created FLINK-35248:
-
Summary: Support table/column comment to modify DDL
Key: FLINK-35248
URL: https://issues.apache.org/jira/browse/FLINK-35248
Project: Flink
Issue Type: New Feature
Components: Flink CDC
Reporter: melin
Table and column comment changes are synchronized to downstream tables：
{code:java}
alter table orders comment '订单'; alter table orders modify address
varchar(1024) null comment '地址'; {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] support op_type metadata columns [flink-cdc]
wangqinghuan closed pull request #1071: support op_type metadata columns URL: https://github.com/apache/flink-cdc/pull/1071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33211][table] support flink table lineage [flink]
HuangZhenQiu commented on code in PR #24618: URL: https://github.com/apache/flink/pull/24618#discussion_r1581716086 ## flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java: ## @@ -133,6 +139,21 @@ protected Transformation translateToPlanInternal( outputTypeInfo, sourceParallelism, sourceParallelismConfigured); + +LineageDataset tableLineageDataset = Review Comment: Thanks for the idea proposed. Yes, I think read the lineage info from connector totally makes sense. If the connector doesn't implement LineageVertexProvider, then we will leave namespace as empty here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33211][table] support flink table lineage [flink]
HuangZhenQiu commented on code in PR #24618:
URL: https://github.com/apache/flink/pull/24618#discussion_r1581321727
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/PhysicalTransformation.java:
##
@@ -34,6 +35,7 @@
public abstract class PhysicalTransformation extends Transformation {
private boolean supportsConcurrentExecutionAttempts = true;
+private LineageVertex lineageVertex;
Review Comment:
Make sense. Added an TransformationWithLineage class for this purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33211][table] support flink table lineage [flink]
HuangZhenQiu commented on code in PR #24618:
URL: https://github.com/apache/flink/pull/24618#discussion_r1581715242
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java:
##
@@ -123,6 +124,7 @@ public class StreamGraph implements Pipeline {
private CheckpointStorage checkpointStorage;
private Set> iterationSourceSinkPairs;
private InternalTimeServiceManager.Provider timerServiceProvider;
+private LineageGraph lineageGraph;
Review Comment:
As we discussed offline, we will keep it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35228][Connectors/Kafka] Fix: DynamicKafkaSource does not read re-added topic for the same cluster [flink-connector-kafka]
mas-chen commented on code in PR #97:
URL:
https://github.com/apache/flink-connector-kafka/pull/97#discussion_r1581372979
##
flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumeratorTest.java:
##
@@ -464,6 +466,87 @@ public void testSnapshotState() throws Throwable {
}
}
+@Test
+public void testSnapshotStateMigration() throws Throwable {
Review Comment:
nit:
```suggestion
public void testEnumeratorStateDoesNotContainStaleTopicPartitions()
throws Throwable {
```
or something similar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35247) Upgrade spotless apply to `2.41.1` in flink-connector-parent to work with Java 21
[ https://issues.apache.org/jira/browse/FLINK-35247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mason Chen updated FLINK-35247: --- Description: Spotless apply version from flink-connector-parent does not work with Java 21 Issue found here: [https://github.com/apache/flink-connector-kafka/pull/98] This is already fixed by spotless apply: [https://github.com/diffplug/spotless/pull/1920] but also requires an upgrade to a later `google-java-format` was: Spotless apply version from flink-connector-parent does not work with Java 21 Tested here: [https://github.com/apache/flink-connector-kafka/pull/98] This is already fixed by spotless apply: https://github.com/diffplug/spotless/pull/1920 > Upgrade spotless apply to `2.41.1` in flink-connector-parent to work with > Java 21 > - > > Key: FLINK-35247 > URL: https://issues.apache.org/jira/browse/FLINK-35247 > Project: Flink > Issue Type: Improvement > Components: Build System / CI, Connectors / Common >Affects Versions: connector-parent-1.1.0 >Reporter: Mason Chen >Priority: Major > > Spotless apply version from flink-connector-parent does not work with Java 21 > Issue found here: [https://github.com/apache/flink-connector-kafka/pull/98] > This is already fixed by spotless apply: > [https://github.com/diffplug/spotless/pull/1920] > but also requires an upgrade to a later `google-java-format` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35247) Upgrade spotless apply to `2.41.1` in flink-connector-parent to work with Java 21
Mason Chen created FLINK-35247: -- Summary: Upgrade spotless apply to `2.41.1` in flink-connector-parent to work with Java 21 Key: FLINK-35247 URL: https://issues.apache.org/jira/browse/FLINK-35247 Project: Flink Issue Type: Improvement Components: Build System / CI, Connectors / Common Affects Versions: connector-parent-1.1.0 Reporter: Mason Chen Spotless apply version from flink-connector-parent does not work with Java 21 Tested here: [https://github.com/apache/flink-connector-kafka/pull/98] This is already fixed by spotless apply: https://github.com/diffplug/spotless/pull/1920 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33211][table] support flink table lineage [flink]
HuangZhenQiu commented on code in PR #24618: URL: https://github.com/apache/flink/pull/24618#discussion_r1581326630 ## flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLegacyTableSourceScan.java: ## @@ -90,6 +90,7 @@ protected Transformation createConversionTransformationIfNeeded( final RowType outputType = (RowType) getOutputType(); final Transformation transformation; final int[] fieldIndexes = computeIndexMapping(true); + Review Comment: Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33211][table] support flink table lineage [flink]
HuangZhenQiu commented on code in PR #24618:
URL: https://github.com/apache/flink/pull/24618#discussion_r1581321727
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/PhysicalTransformation.java:
##
@@ -34,6 +35,7 @@
public abstract class PhysicalTransformation extends Transformation {
private boolean supportsConcurrentExecutionAttempts = true;
+private LineageVertex lineageVertex;
Review Comment:
Make sense. Added an LineagedTransformation class for this purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841301#comment-17841301 ] Zhongqiang Gong edited comment on FLINK-35240 at 4/26/24 5:11 PM: -- [~afedulov] [~robyoung] I opened a pr to patch this issue. Would like help me review? Thank you~ :) was (Author: JIRAUSER301076): [~afedulov] I opened a pr to patch this issue. Would like help me review? Thank you~ :) > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33210] Cleanup the lineage interface comments [flink]
flinkbot commented on PR #24731: URL: https://github.com/apache/flink/pull/24731#issuecomment-2079751600 ## CI report: * 93207bb3788bdcd5f6cf3657c2feb5d92ddb5871 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841301#comment-17841301 ] Zhongqiang Gong commented on FLINK-35240: - [~afedulov] I opened a pr to patch this issue. Would like help me review? Thank you~ :) > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33210] Cleanup the lineage interface comments [flink]
HuangZhenQiu opened a new pull request, #24731: URL: https://github.com/apache/flink/pull/24731 ## What is the purpose of the change Format the class comments of lineage interface classes ## Brief change log - Remove the unneeded empty line of class comments ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35240][Connectors][format]Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record for csv format [flink]
flinkbot commented on PR #24730: URL: https://github.com/apache/flink/pull/24730#issuecomment-2079743542 ## CI report: * 418a88849ef1c9f850cc80b5f691cfe033ac7c09 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35182] Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink Pulsar connector [flink-connector-pulsar]
GOODBOY008 commented on code in PR #90:
URL:
https://github.com/apache/flink-connector-pulsar/pull/90#discussion_r1581286005
##
flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/SampleData.java:
##
@@ -49,7 +49,7 @@ public Foo() {}
@Override
public String toString() {
-return "" + i + "," + f + "," + (bar == null ? "null" :
bar.toString());
+return i + "," + f + "," + (bar == null ? "null" : bar.toString());
Review Comment:
Because ci failed ,I just modify code to trigger ci. Now,This code had been
reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35240: --- Labels: pull-request-available (was: ) > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35240][Connectors][format]Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record for csv format [flink]
GOODBOY008 opened a new pull request, #24730: URL: https://github.com/apache/flink/pull/24730 Changes: - Disable `FLUSH_AFTER_WRITE_VALUE` - Use `ObjectWriter#writeValue(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator, java.lang.Object)` instead of `ObjectWriter#writeValue(java.io.OutputStream, java.lang.Object)` - Manual flush `JsonGenerator` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31223] sql-client.sh fails to start with ssl enabled [flink]
davidradl commented on PR #22026: URL: https://github.com/apache/flink/pull/22026#issuecomment-2079731646 > > @davidradl Make sense to back port this as we should treat this as a bugfix because sql client previously supported SSL, which is a kind of regresssion. > > If you want, just go ahead. > > Ok will do @reswqa Hi I have had a quick look at the back port it is not straight forward. I forgot to ask for the commits to be squashed; the first 2 commits come in nicely with cherry pick but the 3rd with the 10 files does not. It makes changes to files that are not present at 118 for example flink-table/flink-sql-gateway-api/src/main/java/org/apache/flink/table/gateway/api/endpoint/SqlGatewayEndpointFactory.java. I think that more files need to be backported. Could you advise on what else is required for me to do the backport please, unless you want to take over. kind regards, David. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-27146] [Filesystem] Migrate to Junit5 [flink]
ferenc-csaky commented on code in PR #22789:
URL: https://github.com/apache/flink/pull/22789#discussion_r1581270742
##
flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/writer/GSRecoverableFsDataOutputStreamTest.java:
##
@@ -133,57 +131,57 @@ public void before() {
}
}
-@Test
+@TestTemplate
public void emptyStreamShouldHaveProperPositionAndComponentObjectCount() {
if (empty) {
-assertEquals(0, position);
-assertEquals(0, componentObjectCount);
+assertThat(position).isZero();
+assertThat(componentObjectCount).isOne();
Review Comment:
This should be `.isZero()`, and causes the Azure fail. Missed that in my
prev comment. Pls. fix this, if tests are green, then LGTM :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-31223][sqlgateway] Introduce getFlinkConfigurationOptions to g… [flink]
davidradl closed pull request #24729: [FLINK-31223][sqlgateway] Introduce getFlinkConfigurationOptions to g… URL: https://github.com/apache/flink/pull/24729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[ https://issues.apache.org/jira/browse/FLINK-31223 ]
david radley deleted comment on FLINK-31223:
--
was (Author: JIRAUSER300523):
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 1.18 backport
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.20.0
>
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
Re: [PR] [FLINK-31223][sqlgateway] Introduce getFlinkConfigurationOptions to g… [flink]
davidradl commented on PR #24729: URL: https://github.com/apache/flink/pull/24729#issuecomment-2079655100 incomplete will redo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[
https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841279#comment-17841279
]
david radley edited comment on FLINK-31223 at 4/26/24 3:47 PM:
---
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 1.18 backport
was (Author: JIRAUSER300523):
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 118 backport
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.20.0
>
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
> ~[flink-
[jira] [Commented] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[
https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841279#comment-17841279
]
david radley commented on FLINK-31223:
--
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 118 backport
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.20.0
>
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.j
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[
https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841271#comment-17841271
]
Alexander Fedulov commented on FLINK-35240:
---
[~gongzhongqiang] and [~robyoung] you are right!
Unfortunately the docs of the {{FLUSH_PASSED_TO_STREAM}} are lying about it
only being relevant for the {{flush}} method, whereas it indeed controls
individual flushes from the {{close()}} method
{code:java}
/**
* Feature that specifies that calls to {@link #flush} will cause
* matching flush() to underlying {@link OutputStream}
* or {@link Writer}; if disabled this will not be done.
* Main reason to disable this feature is to prevent flushing at
* generator level, if it is not possible to prevent method being
* called by other code (like ObjectMapper or third
* party libraries).
*
* Feature is enabled by default.
*/
FLUSH_PASSED_TO_STREAM(true){code}
> Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
> -
>
> Key: FLINK-35240
> URL: https://issues.apache.org/jira/browse/FLINK-35240
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
>Reporter: Zhongqiang Gong
>Priority: Minor
> Attachments: image-2024-04-26-00-23-29-975.png,
> image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png,
> image-2024-04-26-17-16-30-293.png, screenshot-1.png
>
>
> *Reproduce：*
> * According to user email:
> https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc
> * !image-2024-04-26-00-23-29-975.png!
> *Analysis：*
> * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per
> record.
> *Solution：*
> * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when
> a record added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-35240: -- Attachment: image-2024-04-26-17-16-20-647.png > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-35240: -- Attachment: image-2024-04-26-17-16-30-293.png > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-35240: -- Attachment: image-2024-04-26-17-16-07-925.png > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841267#comment-17841267 ] Oleksandr Nitavskyi commented on FLINK-35232: - [~galenwarren] thanks. We have reduced the amount of methods to the bare minimum. Reflected in the description: * [maxAttempts|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxAttempts__] * [initialRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getInitialRpcTimeout__] * [rpcTimeoutMultiplier|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getRpcTimeoutMultiplier__] * [maxRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxRpcTimeout__] * [totalTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getTotalTimeout__] Thus Flink user will be able to adjust the total timeout time to the checkpoint timeout, so job does it best before it gave up to commit the data. > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. While setting the params enabled > here reduced read timeouts, we still see 503 errors leading to Flink job > restarts. > Thus, in this ticket, we want to specify additional retry settings as noted > in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We need > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. In particular next settings seems to be the minimum required to > adjust GCS timeout with Job's checkpoint config: > * > [maxAttempts|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxAttempts__] > * > [initialRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getInitialRpcTimeout__] > * > [rpcTimeoutMultiplier|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getRpcTimeoutMultiplier__] > * > [maxRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxRpcTimeout__] > * > [totalTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getTotalTimeout__] > > All of the config options should be optional and the default one should be > used in case some of configs are not provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Nitavskyi updated FLINK-35232: Description: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts. Thus, in this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We need [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. In particular next settings seems to be the minimum required to adjust GCS timeout with Job's checkpoint config: * [maxAttempts|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxAttempts__] * [initialRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getInitialRpcTimeout__] * [rpcTimeoutMultiplier|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getRpcTimeoutMultiplier__] * [maxRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxRpcTimeout__] * [totalTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getTotalTimeout__] All of the config options should be optional and the default one should be used in case some of configs are not provided. was: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts. Thus, in this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We need [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. In particular next settings seems to be the minimum required to adjust GCS timeout with Job's checkpoint config: * maxAttempts * initialRpcTimeout * rpcTimeoutMultiplier * maxRpcTimeout * totalTimeout All of the config options should be optional and the default one should be used in case some of configs are not provided. > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. While setting the params enabled > here reduced read timeouts, we still see 503 errors leading to Flink job > restarts. > Thus, in this ticket, we want to specify additional retry settings as noted > in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We need > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. In particular next settings seems to be the minimum required to > adjust GCS timeout with Job's checkpoint config: > * > [maxAttempts|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxAttempts__] > * > [initialRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getInitialRpcTimeout__] > * > [rpcTimeoutMultiplier|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getRpcTimeoutMultiplier__] > * > [maxRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Nitavskyi updated FLINK-35232: Description: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts. Thus, in this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We need [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. In particular next settings seems to be the minimum required to adjust GCS timeout with Job's checkpoint config: * maxAttempts * initialRpcTimeout * rpcTimeoutMultiplier * maxRpcTimeout * totalTimeout All of the config options should be optional and the default one should be used in case some of configs are not provided. was: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts. Thus, in this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We need [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. In particular next settings seems to be the minimum required to adjust GCS timeout with Job's checkpoint config: ``` maxAttempts initialRpcTimeout rpcTimeoutMultiplier maxRpcTimeout totalTimeout ``` All of the config options should be optional and the default one should be used in case some of configs are not provided. > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. While setting the params enabled > here reduced read timeouts, we still see 503 errors leading to Flink job > restarts. > Thus, in this ticket, we want to specify additional retry settings as noted > in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We need > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. In particular next settings seems to be the minimum required to > adjust GCS timeout with Job's checkpoint config: > * maxAttempts > * initialRpcTimeout > * rpcTimeoutMultiplier > * maxRpcTimeout > * totalTimeout > > All of the config options should be optional and the default one should be > used in case some of configs are not provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35232) Support for retry settings on GCS connector
[ https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Nitavskyi updated FLINK-35232: Description: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts. Thus, in this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We need [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. In particular next settings seems to be the minimum required to adjust GCS timeout with Job's checkpoint config: ``` maxAttempts initialRpcTimeout rpcTimeoutMultiplier maxRpcTimeout totalTimeout ``` All of the config options should be optional and the default one should be used in case some of configs are not provided. was: https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts. Thus, in this ticket, we want to specify additional retry settings as noted in [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] We want [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] methods available for Flink users so that they can customize their deployment. > Support for retry settings on GCS connector > --- > > Key: FLINK-35232 > URL: https://issues.apache.org/jira/browse/FLINK-35232 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Vikas M >Assignee: Ravi Singh >Priority: Major > > https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to > specify transport options in GCS connector. While setting the params enabled > here reduced read timeouts, we still see 503 errors leading to Flink job > restarts. > Thus, in this ticket, we want to specify additional retry settings as noted > in > [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries] > We need > [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods] > methods available for Flink users so that they can customize their > deployment. In particular next settings seems to be the minimum required to > adjust GCS timeout with Job's checkpoint config: > ``` > maxAttempts > initialRpcTimeout > rpcTimeoutMultiplier > maxRpcTimeout > totalTimeout > ``` > All of the config options should be optional and the default one should be > used in case some of configs are not provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35022][Connector/DynamoDB] Add TypeInformed DDB Element Converter [flink-connector-aws]
vahmed-hamdy commented on code in PR #136:
URL:
https://github.com/apache/flink-connector-aws/pull/136#discussion_r1581114799
##
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/sink/DynamoDbTypeInformedElementConverter.java:
##
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.BasicArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.NumericTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.CompositeType;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.api.java.tuple.Tuple;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.api.java.typeutils.PojoTypeInfo;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import
org.apache.flink.connector.dynamodb.table.converter.ArrayAttributeConverterProvider;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeConverter;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeConverterProvider;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeValueType;
+import software.amazon.awssdk.enhanced.dynamodb.EnhancedType;
+import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
+import
software.amazon.awssdk.enhanced.dynamodb.internal.mapper.BeanAttributeGetter;
+import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.beans.BeanInfo;
+import java.beans.IntrospectionException;
+import java.beans.Introspector;
+import java.beans.PropertyDescriptor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+/**
+ * A {@link ElementConverter} that converts an element to a {@link
DynamoDbWriteRequest} using
+ * TypeInformation provided.
+ */
+@PublicEvolving
+public class DynamoDbTypeInformedElementConverter
+implements ElementConverter {
+private final CompositeType typeInfo;
+private final boolean ignoreNulls;
+private TableSchema tableSchema;
+
+/**
+ * Creates a {@link DynamoDbTypeInformedElementConverter} that converts an
element to a {@link
+ * DynamoDbWriteRequest} using the provided {@link CompositeType}. Usage:
{@code new
+ *
DynamoDbTypeInformedElementConverter<>(TypeInformation.of(MyPojoClass.class))}
+ *
+ * @param typeInfo The {@link CompositeType} that provides the type
information for the element.
+ */
+public DynamoDbTypeInformedElementConverter(CompositeType
typeInfo) {
+this(typeInfo, true);
+}
+
+public DynamoDbTypeInformedElementConverter(
+CompositeType typeInfo, boolean ignoreNulls) {
+this.typeInfo = typeInfo;
+this.ignoreNulls = ignoreNulls;
+}
+
+@Override
+public void open(Sink.InitContext context) {
+try {
+tableSchema = createTableSchema(typeInfo);
+} catch (IntrospectionException | IllegalStateException |
IllegalArgumentException e) {
+throw new FlinkRuntimeException("Failed to extract DynamoDb table
schema", e);
+}
+}
+
+@Override
+public DynamoDbWriteRequest apply(inputT input, SinkWriter.Context
context) {
+Preconditions.checkNotNull(tableSchema, "TableSchema is not
initialized");
+try {
+return DynamoDbWriteRequest.builder()
+.setType(DynamoDbWriteRequestType.PUT)
+
Re: [PR] [FLINK-35228][Connectors/Kafka] Fix: DynamicKafkaSource does not read re-added topic for the same cluster [flink-connector-kafka]
IgnasD commented on PR #97: URL: https://github.com/apache/flink-connector-kafka/pull/97#issuecomment-2079488271 Changed the title as requested. Also, I've added filtering to `unassignedInitialPartitions` as suggested and covered it with a test case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35228][Connectors/Kafka] Fix DynamicKafkaSourceEnumerator removed topics state cleanup [flink-connector-kafka]
IgnasD commented on code in PR #97:
URL:
https://github.com/apache/flink-connector-kafka/pull/97#discussion_r1581103075
##
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumerator.java:
##
@@ -288,36 +288,39 @@ private void
onHandleSubscribedStreamsFetch(Set fetchedKafkaStreams
// create enumerators
for (Entry> activeClusterTopics :
latestClusterTopicsMap.entrySet()) {
-final Set activeTopicPartitions = new HashSet<>();
-
-if (dynamicKafkaSourceEnumState
+KafkaSourceEnumState kafkaSourceEnumState =
+dynamicKafkaSourceEnumState
.getClusterEnumeratorStates()
-.get(activeClusterTopics.getKey())
-!= null) {
-Set oldTopicPartitions =
-dynamicKafkaSourceEnumState
-.getClusterEnumeratorStates()
-.get(activeClusterTopics.getKey())
-.assignedPartitions();
+.get(activeClusterTopics.getKey());
+
+final KafkaSourceEnumState newKafkaSourceEnumState;
+if (kafkaSourceEnumState != null) {
+Set activeTopicPartitions = new HashSet<>();
+Set oldTopicPartitions =
kafkaSourceEnumState.assignedPartitions();
// filter out removed topics
for (TopicPartition oldTopicPartition : oldTopicPartitions) {
if
(activeClusterTopics.getValue().contains(oldTopicPartition.topic())) {
activeTopicPartitions.add(oldTopicPartition);
}
}
+
+newKafkaSourceEnumState =
+new KafkaSourceEnumState(
+activeTopicPartitions,
+
kafkaSourceEnumState.unassignedInitialPartitions(),
Review Comment:
Added filtering and a unit test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35022][Connector/DynamoDB] Add TypeInformed DDB Element Converter [flink-connector-aws]
vahmed-hamdy commented on code in PR #136:
URL:
https://github.com/apache/flink-connector-aws/pull/136#discussion_r1581083313
##
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/sink/DynamoDbTypeInformedElementConverter.java:
##
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.BasicArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.NumericTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.CompositeType;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.api.java.tuple.Tuple;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.api.java.typeutils.PojoTypeInfo;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import
org.apache.flink.connector.dynamodb.table.converter.ArrayAttributeConverterProvider;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeConverter;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeConverterProvider;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeValueType;
+import software.amazon.awssdk.enhanced.dynamodb.EnhancedType;
+import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
+import
software.amazon.awssdk.enhanced.dynamodb.internal.mapper.BeanAttributeGetter;
+import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.beans.BeanInfo;
+import java.beans.IntrospectionException;
+import java.beans.Introspector;
+import java.beans.PropertyDescriptor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+/**
+ * A {@link ElementConverter} that converts an element to a {@link
DynamoDbWriteRequest} using
+ * TypeInformation provided.
+ */
+@PublicEvolving
+public class DynamoDbTypeInformedElementConverter
Review Comment:
I agree, I would suggest `DefaultDynamoDbElementConverter` since we are
following up with making it default and changing it to optional? wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841232#comment-17841232 ] Zhongqiang Gong edited comment on FLINK-35240 at 4/26/24 1:34 PM: -- !screenshot-1.png! [~afedulov] The close methed always be invorked, in close methed will read *AUTO_CLOSE_TARGET* to determine whether to close or not . was (Author: JIRAUSER301076): !screenshot-1.png! The close methed always be invorked, in close methed will read *AUTO_CLOSE_TARGET* to determine whether to close or not . > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841238#comment-17841238 ] Zhongqiang Gong commented on FLINK-35240: - Base on the logic of _writer. close，We can disable FLUSH_PASSED_TO_STREAM too. So we can control flush in CsvBulkWriter. > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841232#comment-17841232 ] Zhongqiang Gong commented on FLINK-35240: - !截图-1.png! The close methed always be invorked, in close methed will read *AUTO_CLOSE_TARGET* to determine whether to close or not . > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841232#comment-17841232 ] Zhongqiang Gong edited comment on FLINK-35240 at 4/26/24 1:22 PM: -- !screenshot-1.png! The close methed always be invorked, in close methed will read *AUTO_CLOSE_TARGET* to determine whether to close or not . was (Author: JIRAUSER301076): !截图-1.png! The close methed always be invorked, in close methed will read *AUTO_CLOSE_TARGET* to determine whether to close or not . > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35246) SqlClientSSLTest.testGatewayMode failed in AZP
[
https://issues.apache.org/jira/browse/FLINK-35246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841227#comment-17841227
]
Weijie Guo commented on FLINK-35246:
master via 4e6dbe2d1a225a0d0e48fd0997c1f11317402e42.
> SqlClientSSLTest.testGatewayMode failed in AZP
> --
>
> Key: FLINK-35246
> URL: https://issues.apache.org/jira/browse/FLINK-35246
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> {code:java}
> Apr 26 01:51:10 java.lang.IllegalArgumentException: The given host:port
> ('localhost/:36112') doesn't contain a valid port
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.validateHostPortString(NetUtils.java:120)
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.getCorrectHostnamePort(NetUtils.java:81)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayAddress(CliOptionsParser.java:325)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayModeClient(CliOptionsParser.java:296)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:207)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientTestBase.runSqlClient(SqlClientTestBase.java:111)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientSSLTest.testGatewayMode(SqlClientSSLTest.java:74)
> Apr 26 01:51:10 at
> java.base/java.lang.reflect.Method.invoke(Method.java:580)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59173&view=logs&j=26b84117-e436-5720-913e-3e280ce55cae&t=77cc7e77-39a0-5007-6d65-4137ac13a471&l=12418
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-35246) SqlClientSSLTest.testGatewayMode failed in AZP
[
https://issues.apache.org/jira/browse/FLINK-35246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo closed FLINK-35246.
--
Resolution: Fixed
> SqlClientSSLTest.testGatewayMode failed in AZP
> --
>
> Key: FLINK-35246
> URL: https://issues.apache.org/jira/browse/FLINK-35246
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> {code:java}
> Apr 26 01:51:10 java.lang.IllegalArgumentException: The given host:port
> ('localhost/:36112') doesn't contain a valid port
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.validateHostPortString(NetUtils.java:120)
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.getCorrectHostnamePort(NetUtils.java:81)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayAddress(CliOptionsParser.java:325)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayModeClient(CliOptionsParser.java:296)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:207)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientTestBase.runSqlClient(SqlClientTestBase.java:111)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientSSLTest.testGatewayMode(SqlClientSSLTest.java:74)
> Apr 26 01:51:10 at
> java.base/java.lang.reflect.Method.invoke(Method.java:580)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59173&view=logs&j=26b84117-e436-5720-913e-3e280ce55cae&t=77cc7e77-39a0-5007-6d65-4137ac13a471&l=12418
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35246][test] Fix incorrect address construction in SqlClientSSLTest [flink]
reswqa merged PR #24727: URL: https://github.com/apache/flink/pull/24727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35246][test] Fix incorrect address construction in SqlClientSSLTest [flink]
reswqa commented on PR #24727: URL: https://github.com/apache/flink/pull/24727#issuecomment-2079379345 Thanks for the review, merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34629] Fix non-partition topic subscribe lost. [flink-connector-pulsar]
syhily commented on PR #84: URL: https://github.com/apache/flink-connector-pulsar/pull/84#issuecomment-2079360052 Plz close this PR for it has been fixed in #85. @tisonkun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35182] Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink Pulsar connector [flink-connector-pulsar]
syhily commented on code in PR #90:
URL:
https://github.com/apache/flink-connector-pulsar/pull/90#discussion_r1581009843
##
flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/SampleData.java:
##
@@ -49,7 +49,7 @@ public Foo() {}
@Override
public String toString() {
-return "" + i + "," + f + "," + (bar == null ? "null" :
bar.toString());
+return i + "," + f + "," + (bar == null ? "null" : bar.toString());
Review Comment:
I prefer to do only one thing in one PR. This refactor shouldn't be included
in the version bump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35196] [Connector / Pulsar] Fix bouncycastle class not found [flink-connector-pulsar]
syhily commented on PR #91: URL: https://github.com/apache/flink-connector-pulsar/pull/91#issuecomment-2079354340 > > The pulsar-client-all should be upgrade to the 3.0.4 in the mean time. > > @syhily After `pulsar-client-all` was updated to 3.0.4, the unit test failed. I took a brief look and found that `publishTimeStopCursor` seemed to be connected to the consumer and then seek position, so consumer will be disconnect.Then after reconnecting the consumer, an exception of `Exclusive consumer is already connected` occurred. I think this requires more detailed investigation shows that this may be due to some changes in the internal implementation of pulsar-client-all due to version upgrades, and troubleshooting is not that easy. > > So I think it is worth using a separate PR to upgrade pulsar-client-all to version 3.0.4 and solve the problems caused by the related upgrade. > > This PR only solves the problem that the table api cannot find the bouncycastle class. > > Related error logs: [apache/flink-connector-pulsar/actions/runs/8832359197/job/24249517876?pr=91](https://github.com/apache/flink-connector-pulsar/actions/runs/8832359197/job/24249517876?pr=91) > > PTAL. Thanks. Thanks for your works. I think we can divide the work into two steps. Bump pulsar-client-all do require a lot of works. But since the pulsar-client-all depends on the legacy 15on. I think we can exclude these dependencies first in this PR and use 18on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35194][table] Support describe job with job id [flink]
flinkbot commented on PR #24728: URL: https://github.com/apache/flink/pull/24728#issuecomment-2079305845 ## CI report: * 6511c8930cde9f8a3e3acef78fc3bcecb5e9cb51 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31223] sql-client.sh fails to start with ssl enabled [flink]
davidradl commented on PR #22026: URL: https://github.com/apache/flink/pull/22026#issuecomment-2079304678 > @davidradl Make sense to back port this as we should treat this as a bugfix because sql client previously supported SSL, which is a kind of regresssion. > > If you want, just go ahead. Ok will do -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35194) Support describe job syntax and execution
[
https://issues.apache.org/jira/browse/FLINK-35194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35194:
---
Labels: pull-request-available (was: )
> Support describe job syntax and execution
> -
>
> Key: FLINK-35194
> URL: https://issues.apache.org/jira/browse/FLINK-35194
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: xuyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> {code:java}
> { DESCRIBE | DESC } JOB 'xxx'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-35194][table] Support describe job with job id [flink]
xuyangzhong opened a new pull request, #24728:
URL: https://github.com/apache/flink/pull/24728
## What is the purpose of the change
Support syntax '{ DESCRIBE | DESC } JOB 'xxx''
## Brief change log
- *Support to parse syntax { DESCRIBE | DESC } JOB 'xxx'*
- *Support to execute describe job in sql gateway*
- *Add tests*
## Verifying this change
Tests are added to verify it.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? later a single pr will be
introduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-35097) Table API Filesystem connector with 'raw' format repeats last line
[
https://issues.apache.org/jira/browse/FLINK-35097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841197#comment-17841197
]
David Perkins commented on FLINK-35097:
---
Can the fix get applied to 1.17 also?
> Table API Filesystem connector with 'raw' format repeats last line
> --
>
> Key: FLINK-35097
> URL: https://issues.apache.org/jira/browse/FLINK-35097
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.17.1
> Environment: I ran the above test with 1.17.1. I checked for existing
> bug tickets and release notes, but did not find anything, so assuming this
> effects 1.18 and 1.19.
>Reporter: David Perkins
>Assignee: Kumar Mallikarjuna
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0, 1.19.1
>
>
> When using the Filesystem connector with 'raw' format to read text data that
> contains new lines, a row is returned for every line, but always contains the
> contents of the last line.
> For example, with the following file.
> {quote}
> line 1
> line 2
> line 3
> {quote}
> And table definition
> {quote}
> create TABLE MyRawTable (
> `doc` string,
> ) WITH (
> 'path' = 'file:///path/to/data',
> 'format' = 'raw',
>'connector' = 'filesystem'
> );
> {quote}
> Selecting `*` from the table produces three rows all with "line 3" for `doc`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35119][cdc-runtime] Change DataChangeEvent serialization and eserialization [flink-cdc]
yuxiqian commented on code in PR #3226:
URL: https://github.com/apache/flink-cdc/pull/3226#discussion_r1580840112
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/event/DataChangeEventSerializer.java:
##
@@ -76,28 +72,18 @@ public DataChangeEvent deserialize(DataInputView source)
throws IOException {
OperationType op = opSerializer.deserialize(source);
TableId tableId = tableIdSerializer.deserialize(source);
+RecordData before = recordDataSerializer.deserialize(source);
+RecordData after = recordDataSerializer.deserialize(source);
+Map meta = metaSerializer.deserialize(source);
Review Comment:
Is it safe to deserialize here if `op` isn't any known following one? Maybe
we can validate op in advance, and avoid deserializing if an unknown op
occurred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (FLINK-35026) Introduce async execution configurations
[ https://issues.apache.org/jira/browse/FLINK-35026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei resolved FLINK-35026. Resolution: Resolved > Introduce async execution configurations > > > Key: FLINK-35026 > URL: https://issues.apache.org/jira/browse/FLINK-35026 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration, Runtime / Task >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35026) Introduce async execution configurations
[ https://issues.apache.org/jira/browse/FLINK-35026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841182#comment-17841182 ] Yanfei Lei commented on FLINK-35026: Merged into master via 713c30f..3ff2ba4 > Introduce async execution configurations > > > Key: FLINK-35026 > URL: https://issues.apache.org/jira/browse/FLINK-35026 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration, Runtime / Task >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
fredia merged PR #24667: URL: https://github.com/apache/flink/pull/24667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-27146] [Filesystem] Migrate to Junit5 [flink]
kottmann commented on PR #22789: URL: https://github.com/apache/flink/pull/22789#issuecomment-2079230716 Sorry for the delay on my end, I added a new commit to address all your comments above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Kafka part required for Flink Apicurio Avro support. Prototype for review [flink-connector-kafka]
boring-cyborg[bot] commented on PR #99: URL: https://github.com/apache/flink-connector-kafka/pull/99#issuecomment-2079138385 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35119][cdc-runtime] Change DataChangeEvent serialization and eserialization [flink-cdc]
yuxiqian commented on code in PR #3226:
URL: https://github.com/apache/flink-cdc/pull/3226#discussion_r1580840112
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/event/DataChangeEventSerializer.java:
##
@@ -76,28 +72,18 @@ public DataChangeEvent deserialize(DataInputView source)
throws IOException {
OperationType op = opSerializer.deserialize(source);
TableId tableId = tableIdSerializer.deserialize(source);
+RecordData before = recordDataSerializer.deserialize(source);
+RecordData after = recordDataSerializer.deserialize(source);
+Map meta = metaSerializer.deserialize(source);
Review Comment:
Is it safe to deserialize here if `op` isn't any known following one? Or we
can validate op in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35119][cdc-runtime] Change DataChangeEvent serialization and eserialization [flink-cdc]
yuxiqian commented on code in PR #3226:
URL: https://github.com/apache/flink-cdc/pull/3226#discussion_r1580840112
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/event/DataChangeEventSerializer.java:
##
@@ -76,28 +72,18 @@ public DataChangeEvent deserialize(DataInputView source)
throws IOException {
OperationType op = opSerializer.deserialize(source);
TableId tableId = tableIdSerializer.deserialize(source);
+RecordData before = recordDataSerializer.deserialize(source);
+RecordData after = recordDataSerializer.deserialize(source);
+Map meta = metaSerializer.deserialize(source);
Review Comment:
Is it safe to deserialize here if `op` isn't valid? Or we can validate op in
advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-34694) Delete num of associations for streaming outer join
[ https://issues.apache.org/jira/browse/FLINK-34694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17837205#comment-17837205 ] Roman Boyko edited comment on FLINK-34694 at 4/26/24 10:04 AM: --- Hi [~xu_shuai_] ! I prepared and executed all nexmark which uses streaming join (q4, q9 and q20). Because all of them use INNER JOIN (but this optimization works only for outer join) I created the copy with FULL OUTER JOIN for every one. BEFORE optimization: !image-2024-04-26-16-55-19-800.png! AFTER optimization: !image-2024-04-26-16-55-56-994.png! As you can see here - for all INNER JOIN queries the result remains almost the same (small difference most probably cause the measurement error). But for all FULL OUTER JOIN benchmarks the performance is increased. Especially for q20_outer where it was more than 3 times better. The reason of such huge difference can be found on flame graph: BEFORE optimization: !image-2024-04-15-19-15-23-010.png! AFTER optimization: !image-2024-04-15-19-14-41-909.png! Because of prevalence of state.update operation in before-optimization case the rocksdb CompactionJob is invoked more often spending the most CPU time. Totally the performance boost is 6.75 / 5.15 = 1.31 (30%). was (Author: rovboyko): Hi [~xu_shuai_] ! I prepared and executed all nexmark which uses streaming join (q4, q9 and q20). Because all of them use INNER JOIN (but this optimization works only for outer join) I created the copy with FULL OUTER JOIN for every one. BEFORE optimization: !image-2024-04-26-16-55-19-800.png! AFTER optimization: !image-2024-04-26-16-55-56-994.png! As you can see here - for all INNER JOIN queries the result remains almost the same (small difference most probably cause the measurement error). But for all FULL OUTER JOIN benchmarks the performance is increased. Especially for q20_outer where it was more than 3 times better. The reason of such huge difference can be found on flame graph: BEFORE optimization: !image-2024-04-15-19-15-23-010.png! AFTER optimization: !image-2024-04-15-19-14-41-909.png! Because of prevalence of state.update operation in before-optimization case the rocksdb CompactionJob is invoked more often spending the most CPU time. > Delete num of associations for streaming outer join > --- > > Key: FLINK-34694 > URL: https://issues.apache.org/jira/browse/FLINK-34694 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Reporter: Roman Boyko >Priority: Major > Attachments: image-2024-03-15-19-51-29-282.png, > image-2024-03-15-19-52-24-391.png, image-2024-04-15-15-45-51-027.png, > image-2024-04-15-15-46-17-671.png, image-2024-04-15-19-14-14-735.png, > image-2024-04-15-19-14-41-909.png, image-2024-04-15-19-15-23-010.png, > image-2024-04-26-16-55-19-800.png, image-2024-04-26-16-55-56-994.png > > > Currently in StreamingJoinOperator (non-window) in case of OUTER JOIN the > OuterJoinRecordStateView is used to store additional field - the number of > associations for every record. This leads to store additional Tuple2 and > Integer data for every record in outer state. > This functionality is used only for sending: > * -D[nullPaddingRecord] in case of first Accumulate record > * +I[nullPaddingRecord] in case of last Revoke record > The overhead of storing additional data and updating the counter for > associations can be avoided by checking the input state for these events. > > The proposed solution can be found here - > [https://github.com/rovboyko/flink/commit/1ca2f5bdfc2d44b99d180abb6a4dda123e49d423] > > According to the nexmark q20 test (changed to OUTER JOIN) it could increase > the performance up to 20%: > * Before: > !image-2024-03-15-19-52-24-391.png! > * After: > !image-2024-03-15-19-51-29-282.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34694) Delete num of associations for streaming outer join
[ https://issues.apache.org/jira/browse/FLINK-34694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17837205#comment-17837205 ] Roman Boyko edited comment on FLINK-34694 at 4/26/24 10:05 AM: --- Hi [~xu_shuai_] ! I prepared and executed all nexmark which uses streaming join (q4, q9 and q20). Because all of them use INNER JOIN (but this optimization works only for outer join) I created the copy with FULL OUTER JOIN for every one. BEFORE optimization: !image-2024-04-26-16-55-19-800.png! AFTER optimization: !image-2024-04-26-16-55-56-994.png! As you can see here - for all INNER JOIN queries the result remains almost the same (small difference most probably cause the measurement error). But for all FULL OUTER JOIN benchmarks the performance is increased. Especially for q20_outer where it was more than 3 times better. The reason of such huge difference can be found on flame graph: BEFORE optimization: !image-2024-04-15-19-15-23-010.png! AFTER optimization: !image-2024-04-15-19-14-41-909.png! Because of prevalence of state.update operation in before-optimization case the rocksdb CompactionJob is invoked more often spending the most CPU time. Totally the performance boost is 6.75 / 5.15 = 1.31 (31%). was (Author: rovboyko): Hi [~xu_shuai_] ! I prepared and executed all nexmark which uses streaming join (q4, q9 and q20). Because all of them use INNER JOIN (but this optimization works only for outer join) I created the copy with FULL OUTER JOIN for every one. BEFORE optimization: !image-2024-04-26-16-55-19-800.png! AFTER optimization: !image-2024-04-26-16-55-56-994.png! As you can see here - for all INNER JOIN queries the result remains almost the same (small difference most probably cause the measurement error). But for all FULL OUTER JOIN benchmarks the performance is increased. Especially for q20_outer where it was more than 3 times better. The reason of such huge difference can be found on flame graph: BEFORE optimization: !image-2024-04-15-19-15-23-010.png! AFTER optimization: !image-2024-04-15-19-14-41-909.png! Because of prevalence of state.update operation in before-optimization case the rocksdb CompactionJob is invoked more often spending the most CPU time. Totally the performance boost is 6.75 / 5.15 = 1.31 (30%). > Delete num of associations for streaming outer join > --- > > Key: FLINK-34694 > URL: https://issues.apache.org/jira/browse/FLINK-34694 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Reporter: Roman Boyko >Priority: Major > Attachments: image-2024-03-15-19-51-29-282.png, > image-2024-03-15-19-52-24-391.png, image-2024-04-15-15-45-51-027.png, > image-2024-04-15-15-46-17-671.png, image-2024-04-15-19-14-14-735.png, > image-2024-04-15-19-14-41-909.png, image-2024-04-15-19-15-23-010.png, > image-2024-04-26-16-55-19-800.png, image-2024-04-26-16-55-56-994.png > > > Currently in StreamingJoinOperator (non-window) in case of OUTER JOIN the > OuterJoinRecordStateView is used to store additional field - the number of > associations for every record. This leads to store additional Tuple2 and > Integer data for every record in outer state. > This functionality is used only for sending: > * -D[nullPaddingRecord] in case of first Accumulate record > * +I[nullPaddingRecord] in case of last Revoke record > The overhead of storing additional data and updating the counter for > associations can be avoided by checking the input state for these events. > > The proposed solution can be found here - > [https://github.com/rovboyko/flink/commit/1ca2f5bdfc2d44b99d180abb6a4dda123e49d423] > > According to the nexmark q20 test (changed to OUTER JOIN) it could increase > the performance up to 20%: > * Before: > !image-2024-03-15-19-52-24-391.png! > * After: > !image-2024-03-15-19-51-29-282.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34694) Delete num of associations for streaming outer join
[ https://issues.apache.org/jira/browse/FLINK-34694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17837205#comment-17837205 ] Roman Boyko edited comment on FLINK-34694 at 4/26/24 10:01 AM: --- Hi [~xu_shuai_] ! I prepared and executed all nexmark which uses streaming join (q4, q9 and q20). Because all of them use INNER JOIN (but this optimization works only for outer join) I created the copy with FULL OUTER JOIN for every one. BEFORE optimization: !image-2024-04-26-16-55-19-800.png! AFTER optimization: !image-2024-04-26-16-55-56-994.png! As you can see here - for all INNER JOIN queries the result remains almost the same (small difference most probably cause the measurement error). But for all FULL OUTER JOIN benchmarks the performance is increased. Especially for q20_outer where it was more than 3 times better. The reason of such huge difference can be found on flame graph: BEFORE optimization: !image-2024-04-15-19-15-23-010.png! AFTER optimization: !image-2024-04-15-19-14-41-909.png! Because of prevalence of state.update operation in before-optimization case the rocksdb CompactionJob is invoked more often spending the most CPU time. was (Author: rovboyko): Hi [~xu_shuai_] ! I prepared and executed all nexmark which uses streaming join (q4, q7, q9 and q20). Because all of them use INNER JOIN (but this optimization works only for outer join) I created the copy with FULL OUTER JOIN for every one. BEFORE optimization: !image-2024-04-15-15-45-51-027.png! AFTER optimization: !image-2024-04-15-15-46-17-671.png! As you can see here - for all queries except q20_outer the result remains almost the same (small difference most probably cause the measurement error). But for q20_outer the performance is more than 2 times better (I repeated the test several times). The reason of such huge difference can be found on flame graph: BEFORE optimization: !image-2024-04-15-19-15-23-010.png! AFTER optimization: !image-2024-04-15-19-14-41-909.png! Because of prevalence of state.update operation in before-optimization case the rocksdb CompactionJob is invoked more often spending the most CPU time. There is no such performance boost for q4, q7 and q9 because: * q7 translates to Interval join * q4 and q9 transformed to InnerJoin by FlinkFilterJoinRule (maybe this is a bug, I will check later) > Delete num of associations for streaming outer join > --- > > Key: FLINK-34694 > URL: https://issues.apache.org/jira/browse/FLINK-34694 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Reporter: Roman Boyko >Priority: Major > Attachments: image-2024-03-15-19-51-29-282.png, > image-2024-03-15-19-52-24-391.png, image-2024-04-15-15-45-51-027.png, > image-2024-04-15-15-46-17-671.png, image-2024-04-15-19-14-14-735.png, > image-2024-04-15-19-14-41-909.png, image-2024-04-15-19-15-23-010.png, > image-2024-04-26-16-55-19-800.png, image-2024-04-26-16-55-56-994.png > > > Currently in StreamingJoinOperator (non-window) in case of OUTER JOIN the > OuterJoinRecordStateView is used to store additional field - the number of > associations for every record. This leads to store additional Tuple2 and > Integer data for every record in outer state. > This functionality is used only for sending: > * -D[nullPaddingRecord] in case of first Accumulate record > * +I[nullPaddingRecord] in case of last Revoke record > The overhead of storing additional data and updating the counter for > associations can be avoided by checking the input state for these events. > > The proposed solution can be found here - > [https://github.com/rovboyko/flink/commit/1ca2f5bdfc2d44b99d180abb6a4dda123e49d423] > > According to the nexmark q20 test (changed to OUTER JOIN) it could increase > the performance up to 20%: > * Before: > !image-2024-03-15-19-52-24-391.png! > * After: > !image-2024-03-15-19-51-29-282.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35192) operator oom
[ https://issues.apache.org/jira/browse/FLINK-35192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841129#comment-17841129 ] Biao Geng commented on FLINK-35192: --- !screenshot-3.png! According to the flink k8s op's codes, the deleteOnExit() is called when create config files or pod template files. It looks like that it is possible to lead the memory leak if the operator pod runs for a long time. In the operator's FlinkConfigManager implementation, we would clean up these temp files/dirs. Maybe we can safely remove the deleteOnExit() usage? cc [~gyfora] Also, from the attached yaml, it looks like a custom flink k8s op image(gdc-flink-kubernetes-operator:1.6.1-GDC1.0.2) is used. [~stupid_pig] would you mind checking if your codes call methods like deleteOnExit if you have some customized changes to the operator? > operator oom > > > Key: FLINK-35192 > URL: https://issues.apache.org/jira/browse/FLINK-35192 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.6.1 > Environment: jdk: openjdk11 > operator version: 1.6.1 >Reporter: chenyuzhi >Priority: Major > Attachments: image-2024-04-22-15-47-49-455.png, > image-2024-04-22-15-52-51-600.png, image-2024-04-22-15-58-23-269.png, > image-2024-04-22-15-58-42-850.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > > The kubernetest operator docker process was killed by kernel cause out of > memory(the time is 2024.04.03: 18:16) > !image-2024-04-22-15-47-49-455.png! > Metrics: > the pod memory (RSS) is increasing slowly in the past 7 days: > !screenshot-1.png! > However the jvm memory metrics of operator not shown obvious anomaly： > !image-2024-04-22-15-58-23-269.png! > !image-2024-04-22-15-58-42-850.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35192) operator oom
[ https://issues.apache.org/jira/browse/FLINK-35192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biao Geng updated FLINK-35192: -- Attachment: screenshot-3.png > operator oom > > > Key: FLINK-35192 > URL: https://issues.apache.org/jira/browse/FLINK-35192 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.6.1 > Environment: jdk: openjdk11 > operator version: 1.6.1 >Reporter: chenyuzhi >Priority: Major > Attachments: image-2024-04-22-15-47-49-455.png, > image-2024-04-22-15-52-51-600.png, image-2024-04-22-15-58-23-269.png, > image-2024-04-22-15-58-42-850.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > > The kubernetest operator docker process was killed by kernel cause out of > memory(the time is 2024.04.03: 18:16) > !image-2024-04-22-15-47-49-455.png! > Metrics: > the pod memory (RSS) is increasing slowly in the past 7 days: > !screenshot-1.png! > However the jvm memory metrics of operator not shown obvious anomaly： > !image-2024-04-22-15-58-23-269.png! > !image-2024-04-22-15-58-42-850.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35195) Support the execution of create materialized table in continuous refresh mode
[
https://issues.apache.org/jira/browse/FLINK-35195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dalongliu reassigned FLINK-35195:
-
Assignee: dalongliu
> Support the execution of create materialized table in continuous refresh mode
> -
>
> Key: FLINK-35195
> URL: https://issues.apache.org/jira/browse/FLINK-35195
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Gateway
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: dalongliu
>Priority: Major
> Fix For: 1.20.0
>
>
> In continuous refresh mode, support creates materialized table and its
> background refresh job:
> {code:SQL}
> CREATE MATERIALIZED TABLE [catalog_name.][db_name.]table_name
>
> [ ([ ]) ]
>
> [COMMENT table_comment]
>
> [PARTITIONED BY (partition_column_name1, partition_column_name2, ...)]
>
> [WITH (key1=val1, key2=val2, ...)]
>
> FRESHNESS = INTERVAL '' { SECOND | MINUTE | HOUR | DAY }
>
> [REFRESH_MODE = { CONTINUOUS | FULL }]
>
> AS
>
> :
> [CONSTRAINT constraint_name] PRIMARY KEY (column_name, ...) NOT ENFORCED
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35189) Introduce test-filesystem Catalog based on FileSystem Connector to support materialized table
[ https://issues.apache.org/jira/browse/FLINK-35189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841128#comment-17841128 ] dalongliu commented on FLINK-35189: --- Merged in master: 714d1cb2e0bd0df03393492dc87cbd800af63e1b > Introduce test-filesystem Catalog based on FileSystem Connector to support > materialized table > - > > Key: FLINK-35189 > URL: https://issues.apache.org/jira/browse/FLINK-35189 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem, Table SQL / API, Tests >Affects Versions: 1.20.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-35189) Introduce test-filesystem Catalog based on FileSystem Connector to support materialized table
[ https://issues.apache.org/jira/browse/FLINK-35189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu resolved FLINK-35189. --- Resolution: Fixed > Introduce test-filesystem Catalog based on FileSystem Connector to support > materialized table > - > > Key: FLINK-35189 > URL: https://issues.apache.org/jira/browse/FLINK-35189 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem, Table SQL / API, Tests >Affects Versions: 1.20.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35189][test-utils]Introduce test-filesystem Catalog based on FileSystem Connector to support materialized table [flink]
lsyldliu merged PR #24712: URL: https://github.com/apache/flink/pull/24712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35197) Support the execution of suspend, resume materialized table in continuous refresh mode
[
https://issues.apache.org/jira/browse/FLINK-35197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dalongliu reassigned FLINK-35197:
-
Assignee: Feng Jin
> Support the execution of suspend, resume materialized table in continuous
> refresh mode
> --
>
> Key: FLINK-35197
> URL: https://issues.apache.org/jira/browse/FLINK-35197
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Gateway
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: Feng Jin
>Priority: Major
> Fix For: 1.20.0
>
>
> In continuous refresh mode, support suspend, resume the background refresh
> job of materialized table.
> {code:SQL}
> // suspend
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name SUSPEND
> // resume
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name RESUME
> [WITH('key1' = 'val1', 'key2' = 'val2')]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35194) Support describe job syntax and execution
[
https://issues.apache.org/jira/browse/FLINK-35194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841122#comment-17841122
]
dalongliu commented on FLINK-35194:
---
Yeah, assigned to you.
> Support describe job syntax and execution
> -
>
> Key: FLINK-35194
> URL: https://issues.apache.org/jira/browse/FLINK-35194
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: xuyang
>Priority: Major
> Fix For: 1.20.0
>
>
> {code:java}
> { DESCRIBE | DESC } JOB 'xxx'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35194) Support describe job syntax and execution
[
https://issues.apache.org/jira/browse/FLINK-35194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dalongliu reassigned FLINK-35194:
-
Assignee: xuyang
> Support describe job syntax and execution
> -
>
> Key: FLINK-35194
> URL: https://issues.apache.org/jira/browse/FLINK-35194
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: xuyang
>Priority: Major
> Fix For: 1.20.0
>
>
> {code:java}
> { DESCRIBE | DESC } JOB 'xxx'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35119][cdc-runtime] Change DataChangeEvent serialization and eserialization [flink-cdc]
yuxiqian commented on PR #3226: URL: https://github.com/apache/flink-cdc/pull/3226#issuecomment-2079004056 @Jiabao-Sun Sure, I'll take it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35119][cdc-runtime] Change DataChangeEvent serialization and eserialization [flink-cdc]
Jiabao-Sun commented on PR #3226: URL: https://github.com/apache/flink-cdc/pull/3226#issuecomment-2078993303 Thanks @zhongqishang for this fix. Hi @yuxiqian, do you have time to review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841117#comment-17841117 ] Alexander Fedulov commented on FLINK-35240: --- I don't think touching any of the flush-specific properties should be necessary. You can see in the FlameGraph that flush calls are due to close being called and, as [~robyoung] mentioned, this is what JsonGenerator.Feature#AUTO_CLOSE_TARGET is there for. > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-32877) Support for HTTP connect and timeout options while writes in GCS connector
[ https://issues.apache.org/jira/browse/FLINK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer resolved FLINK-32877. --- Resolution: Done > Support for HTTP connect and timeout options while writes in GCS connector > -- > > Key: FLINK-32877 > URL: https://issues.apache.org/jira/browse/FLINK-32877 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Jayadeep Jayaraman >Assignee: Ravi Singh >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > The current GCS connector uses the gcs java storage library and bypasses the > hadoop gcs connector which supports multiple http options. There are > situations where GCS takes longer to provide a response for a PUT operation > than the default value. > This change will allow users to customize their connect time and read timeout > based on their application -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32877][Filesystem]add HTTP options to gcs-cloud-storage client [flink]
dannycranmer commented on PR #23226: URL: https://github.com/apache/flink/pull/23226#issuecomment-2078971443 Merged in https://github.com/apache/flink/pull/24673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32877][Filesystem]add HTTP options to gcs-cloud-storage client [flink]
dannycranmer closed pull request #23226: [FLINK-32877][Filesystem]add HTTP options to gcs-cloud-storage client URL: https://github.com/apache/flink/pull/23226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35246][test] Fix incorrect address construction in SqlClientSSLTest [flink]
flinkbot commented on PR #24727: URL: https://github.com/apache/flink/pull/24727#issuecomment-2078971258 ## CI report: * a327b1e5ee7948bfd7f5c699222be5575e72f6f6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35244) Move package for flink-connector-tidb-cdc test
[ https://issues.apache.org/jira/browse/FLINK-35244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun reassigned FLINK-35244: -- Assignee: Xie Yi > Move package for flink-connector-tidb-cdc test > -- > > Key: FLINK-35244 > URL: https://issues.apache.org/jira/browse/FLINK-35244 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xie Yi >Assignee: Xie Yi >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-26-16-19-39-297.png > > > test case for flink-connector-tidb-cdc should under > *org.apache.flink.cdc.connectors.tidb* package > instead of *org.apache.flink.cdc.connectors* > !image-2024-04-26-16-19-39-297.png! > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32877) Support for HTTP connect and timeout options while writes in GCS connector
[ https://issues.apache.org/jira/browse/FLINK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-32877: -- Affects Version/s: 1.18.1 1.19.0 > Support for HTTP connect and timeout options while writes in GCS connector > -- > > Key: FLINK-32877 > URL: https://issues.apache.org/jira/browse/FLINK-32877 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1, 1.19.0, 1.18.1 >Reporter: Jayadeep Jayaraman >Assignee: Ravi Singh >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > The current GCS connector uses the gcs java storage library and bypasses the > hadoop gcs connector which supports multiple http options. There are > situations where GCS takes longer to provide a response for a PUT operation > than the default value. > This change will allow users to customize their connect time and read timeout > based on their application -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32877) Support for HTTP connect and timeout options while writes in GCS connector
[ https://issues.apache.org/jira/browse/FLINK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-32877: -- Fix Version/s: 1.20.0 > Support for HTTP connect and timeout options while writes in GCS connector > -- > > Key: FLINK-32877 > URL: https://issues.apache.org/jira/browse/FLINK-32877 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.3, 1.16.2, 1.17.1 >Reporter: Jayadeep Jayaraman >Assignee: Ravi Singh >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > The current GCS connector uses the gcs java storage library and bypasses the > hadoop gcs connector which supports multiple http options. There are > situations where GCS takes longer to provide a response for a PUT operation > than the default value. > This change will allow users to customize their connect time and read timeout > based on their application -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32877][Filesystem][Rebased] Add HTTP options to java-storage client [flink]
dannycranmer merged PR #24673: URL: https://github.com/apache/flink/pull/24673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32877][Filesystem][Rebased] Add HTTP options to java-storage client [flink]
dannycranmer commented on PR #24673: URL: https://github.com/apache/flink/pull/24673#issuecomment-2078968511 Apologies I did not fix the commit message 🤦 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35246) SqlClientSSLTest.testGatewayMode failed in AZP
[
https://issues.apache.org/jira/browse/FLINK-35246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35246:
---
Labels: pull-request-available (was: )
> SqlClientSSLTest.testGatewayMode failed in AZP
> --
>
> Key: FLINK-35246
> URL: https://issues.apache.org/jira/browse/FLINK-35246
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> {code:java}
> Apr 26 01:51:10 java.lang.IllegalArgumentException: The given host:port
> ('localhost/:36112') doesn't contain a valid port
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.validateHostPortString(NetUtils.java:120)
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.getCorrectHostnamePort(NetUtils.java:81)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayAddress(CliOptionsParser.java:325)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayModeClient(CliOptionsParser.java:296)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:207)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientTestBase.runSqlClient(SqlClientTestBase.java:111)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientSSLTest.testGatewayMode(SqlClientSSLTest.java:74)
> Apr 26 01:51:10 at
> java.base/java.lang.reflect.Method.invoke(Method.java:580)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59173&view=logs&j=26b84117-e436-5720-913e-3e280ce55cae&t=77cc7e77-39a0-5007-6d65-4137ac13a471&l=12418
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-31223] sql-client.sh fails to start with ssl enabled [flink]
reswqa commented on PR #22026: URL: https://github.com/apache/flink/pull/22026#issuecomment-2078960338 @davidradl Make sense to back port this as we should treat this as a bugfix because sql client previously supported SSL, which is a kind of regresssion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-35235) Fix missing dependencies in the uber jar
[ https://issues.apache.org/jira/browse/FLINK-35235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun resolved FLINK-35235. Assignee: LvYanquan Resolution: Fixed Resolved via * cdc master: ec643c9dd7365261f3cee620d4d6bd5d042917e0 * cdc release-3.1: b96ea11cc7df6c3d57a155573f29c18bf9d787ae > Fix missing dependencies in the uber jar > > > Key: FLINK-35235 > URL: https://issues.apache.org/jira/browse/FLINK-35235 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: 3.1.0 >Reporter: LvYanquan >Assignee: LvYanquan >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: image-2024-04-25-15-17-20-987.png, > image-2024-04-25-15-17-34-717.png > > > Some class of Kafka were not included in fat jar. > !image-2024-04-25-15-17-34-717.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32877][Filesystem][Rebased] Add HTTP options to java-storage client [flink]
dannycranmer commented on PR #24673: URL: https://github.com/apache/flink/pull/24673#issuecomment-2078965258 LGTM, I will fix the commit message on merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35194) Support describe job syntax and execution
[
https://issues.apache.org/jira/browse/FLINK-35194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841113#comment-17841113
]
xuyang commented on FLINK-35194:
Hi, can I take this jira?
> Support describe job syntax and execution
> -
>
> Key: FLINK-35194
> URL: https://issues.apache.org/jira/browse/FLINK-35194
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Priority: Major
> Fix For: 1.20.0
>
>
> {code:java}
> { DESCRIBE | DESC } JOB 'xxx'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35235][pipeline-connector][kafka] Fix missing dependencies in the uber jar of Kafka pipeline sink. [flink-cdc]
Jiabao-Sun merged PR #3262: URL: https://github.com/apache/flink-cdc/pull/3262 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35246) SqlClientSSLTest.testGatewayMode failed in AZP
[ https://issues.apache.org/jira/browse/FLINK-35246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo reassigned FLINK-35246: -- Assignee: Weijie Guo > SqlClientSSLTest.testGatewayMode failed in AZP > -- > > Key: FLINK-35246 > URL: https://issues.apache.org/jira/browse/FLINK-35246 > Project: Flink > Issue Type: Bug > Components: Build System / CI >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35246) SqlClientSSLTest.testGatewayMode failed in AZP
Weijie Guo created FLINK-35246: -- Summary: SqlClientSSLTest.testGatewayMode failed in AZP Key: FLINK-35246 URL: https://issues.apache.org/jira/browse/FLINK-35246 Project: Flink Issue Type: Bug Components: Build System / CI Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35246) SqlClientSSLTest.testGatewayMode failed in AZP
[
https://issues.apache.org/jira/browse/FLINK-35246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1784#comment-1784
]
Weijie Guo commented on FLINK-35246:
{code:java}
InetSocketAddress.createUnresolved(
SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetAddress(),
SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetPort())
.toString()
{code}
this construction of InetSocketAddress fails on Java 17 because the toString
representation is not guaranteed to return something of the form host:port.
> SqlClientSSLTest.testGatewayMode failed in AZP
> --
>
> Key: FLINK-35246
> URL: https://issues.apache.org/jira/browse/FLINK-35246
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Major
> Fix For: 1.20.0
>
>
> {code:java}
> Apr 26 01:51:10 java.lang.IllegalArgumentException: The given host:port
> ('localhost/:36112') doesn't contain a valid port
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.validateHostPortString(NetUtils.java:120)
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.getCorrectHostnamePort(NetUtils.java:81)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayAddress(CliOptionsParser.java:325)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayModeClient(CliOptionsParser.java:296)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:207)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientTestBase.runSqlClient(SqlClientTestBase.java:111)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientSSLTest.testGatewayMode(SqlClientSSLTest.java:74)
> Apr 26 01:51:10 at
> java.base/java.lang.reflect.Method.invoke(Method.java:580)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59173&view=logs&j=26b84117-e436-5720-913e-3e280ce55cae&t=77cc7e77-39a0-5007-6d65-4137ac13a471&l=12418
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [BP-3.1][FLINK-35235][pipeline-connector][kafka] Fix missing dependencies in the uber jar of Kafka pipeline sink. [flink-cdc]
Jiabao-Sun merged PR #3263: URL: https://github.com/apache/flink-cdc/pull/3263 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-35246) SqlClientSSLTest.testGatewayMode failed in AZP
[
https://issues.apache.org/jira/browse/FLINK-35246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1784#comment-1784
]
Weijie Guo edited comment on FLINK-35246 at 4/26/24 8:59 AM:
-
{code:java}
InetSocketAddress.createUnresolved(
SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetAddress(),
SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetPort())
.toString()
{code}
The construction of InetSocketAddress fails on Java 17 because the toString
representation is not guaranteed to return something of the form host:port.
was (Author: weijie guo):
{code:java}
InetSocketAddress.createUnresolved(
SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetAddress(),
SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetPort())
.toString()
{code}
this construction of InetSocketAddress fails on Java 17 because the toString
representation is not guaranteed to return something of the form host:port.
> SqlClientSSLTest.testGatewayMode failed in AZP
> --
>
> Key: FLINK-35246
> URL: https://issues.apache.org/jira/browse/FLINK-35246
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Major
> Fix For: 1.20.0
>
>
> {code:java}
> Apr 26 01:51:10 java.lang.IllegalArgumentException: The given host:port
> ('localhost/:36112') doesn't contain a valid port
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.validateHostPortString(NetUtils.java:120)
> Apr 26 01:51:10 at
> org.apache.flink.util.NetUtils.getCorrectHostnamePort(NetUtils.java:81)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayAddress(CliOptionsParser.java:325)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.cli.CliOptionsParser.parseGatewayModeClient(CliOptionsParser.java:296)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:207)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientTestBase.runSqlClient(SqlClientTestBase.java:111)
> Apr 26 01:51:10 at
> org.apache.flink.table.client.SqlClientSSLTest.testGatewayMode(SqlClientSSLTest.java:74)
> Apr 26 01:51:10 at
> java.base/java.lang.reflect.Method.invoke(Method.java:580)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> Apr 26 01:51:10 at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59173&view=logs&j=26b84117-e436-5720-913e-3e280ce55cae&t=77cc7e77-39a0-5007-6d65-4137ac13a471&l=12418
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-32084][checkpoint] Migrate current file merging of channel state snapshot into the unify file merging framework [flink]
fredia commented on PR #24653: URL: https://github.com/apache/flink/pull/24653#issuecomment-2078953534 @Zakelly @1996fanrui would you please to take a look? thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [tidb] Add metrics for tidb connector [flink-cdc]
xieyi888 commented on PR #1974: URL: https://github.com/apache/flink-cdc/pull/1974#issuecomment-2078942188 > add metrcis: currentFetchEventTimeLag, currentEmitEventTimeLag, sourceIdleTime for TiKVRichParallelSourceFunction > Thanks @xieyi888 for the great work! Before this PR could be merged, could you please rebase it with latest `master` branch? Thanks a lot for pushing this PR. As it was created before version 2.3.0 (2022-11-10) I had create new issue and PR to solve it https://issues.apache.org/jira/browse/FLINK-35245 https://github.com/apache/flink-cdc/pull/3266 Please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
