[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii closed pull request #17938: [FLINK-25073][streaming] Introduce TreeMode description for vertices
gaoyunhaii closed pull request #17938: URL: https://github.com/apache/flink/pull/17938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * 4ea3c9f8fd437a0e6d183bfb62101f54f89de34e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29219) * a65b12ce87fe15a5474b0a740e9a054e7685941d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29612) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * 5575d8841ea45e8af178b2bb756c7d410ea82ae4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
Yufei Zhang created FLINK-25686: --- Summary: Support Pulsar Schema evolution in Pulsar Connector Key: FLINK-25686 URL: https://issues.apache.org/jira/browse/FLINK-25686 Project: Flink Issue Type: New Feature Components: Connectors / Pulsar Reporter: Yufei Zhang Currently the source connector reads all messages as Message, and implements a deserialization abstraction to deserialize the byte array message. This deserialization abstraction supports both flink schema and pulsar schema. However because of this extra layer of abstraction, even when using pulsar schema, the schema info is not provided to the consumer (client), so it is not using any pulsar’s schema validation mechanism. (If pulsar client has a schema, it will send the schema to broker on connect and broker will validate if it is a valid schema according to a preset compatibility rule :[ https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477605#comment-17477605 ] Yufei Zhang commented on FLINK-25686: - Please assign this ticket to me [~fpaul] ~ Thank you ! > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * 5575d8841ea45e8af178b2bb756c7d410ea82ae4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * a9175f7c992d1cb30579dc88d4406c228e06a67a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29560) * dcebc46b47336a4f7e41003f091f4c4fb6a28bdc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29605) * 8515ccb41dc778e38a97aa5cd7e80440a9ef389e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29609) * 7a1ba6d584788f1b07bc5bd82d879327fbdc48e6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25073) Introduce Tree Mode description for job vertex
[ https://issues.apache.org/jira/browse/FLINK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao closed FLINK-25073. --- Resolution: Fixed Merged on master via f2d03b0baea08ed59eabb72afad6ebcc4c525036 > Introduce Tree Mode description for job vertex > -- > > Key: FLINK-25073 > URL: https://issues.apache.org/jira/browse/FLINK-25073 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Reporter: Wenlong Lyu >Assignee: Wenlong Lyu >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * a9175f7c992d1cb30579dc88d4406c228e06a67a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29560) * dcebc46b47336a4f7e41003f091f4c4fb6a28bdc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29605) * 8515ccb41dc778e38a97aa5cd7e80440a9ef389e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29609) * 7a1ba6d584788f1b07bc5bd82d879327fbdc48e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29614) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18363: [Flink-25600][table-planner] Support new statement set syntax in sql client and update docs
flinkbot edited a comment on pull request #18363: URL: https://github.com/apache/flink/pull/18363#issuecomment-1012985353 ## CI report: * 023810c97360cbec41e825735054561aa9ed2dcd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29599) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18380: [FLINK-25676][python] Support set_description in DataStream API
flinkbot edited a comment on pull request #18380: URL: https://github.com/apache/flink/pull/18380#issuecomment-1014348700 ## CI report: * 5e49df6104317779a59d57e2e4ecf08d7e36eff4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29600) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #17098: [FLINK-15648][flink-kubernetes] Support to configure limit for CPU & memory
wangyang0918 commented on a change in pull request #17098:
URL: https://github.com/apache/flink/pull/17098#discussion_r786443356
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesJobManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getJobManagerCPU() {
return flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_CPU);
}
+public double getJobManagerCPULimitFactor() {
+double limitFactor =
Review comment:
nit: `limitFactor` could be final.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesJobManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getJobManagerCPU() {
return flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_CPU);
}
+public double getJobManagerCPULimitFactor() {
+double limitFactor =
+
flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_CPU_LIMIT_FACTOR);
+checkArgument(
+limitFactor >= 1,
+"%s should be greater or equal to 1.",
+KubernetesConfigOptions.JOB_MANAGER_CPU_LIMIT_FACTOR.key());
+return limitFactor;
+}
+
+public double getJobManagerMemoryLimitFactor() {
+double limitFactor =
Review comment:
nit: limitFactor could be final.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesTaskManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getTaskManagerCPU() {
.doubleValue();
}
+public double getTaskManagerCPULimitFactor() {
+double limitFactor =
Review comment:
nit: limitFactor could be final.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##
@@ -310,26 +310,33 @@ public static CompletedCheckpointStore
createCompletedCheckpointStore(
*
* @param resourceRequirements resource requirements in pod template
* @param mem Memory in mb.
+ * @param memoryLimitFactor limit factor for the memory, used to set the
limit resources.
* @param cpu cpu.
+ * @param cpuLimitFactor limit factor for the cpu, used to set the limit
resources.
* @param externalResources external resources
* @param externalResourceConfigKeys config keys of external resources
* @return KubernetesResource requirements.
*/
public static ResourceRequirements getResourceRequirements(
ResourceRequirements resourceRequirements,
int mem,
+double memoryLimitFactor,
double cpu,
+double cpuLimitFactor,
Map externalResources,
Map externalResourceConfigKeys) {
final Quantity cpuQuantity = new Quantity(String.valueOf(cpu));
+final Quantity cpuLimitQuantity = new Quantity(String.valueOf(cpu *
cpuLimitFactor));
final Quantity memQuantity = new Quantity(mem +
Constants.RESOURCE_UNIT_MB);
+final Quantity memQuantityLimit =
+new Quantity(((int) (mem * memoryLimitFactor)) +
Constants.RESOURCE_UNIT_MB);
Review comment:
Maybe we do not need to use the `int` here since K8s could also support
decimal amount for memory resource. e.g. `1024.5m` `1.5g`. But the current
implementation also make sense to me.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesTaskManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getTaskManagerCPU() {
.doubleValue();
}
+public double getTaskManagerCPULimitFactor() {
+double limitFactor =
+
flinkConfig.getDouble(KubernetesConfigOptions.TASK_MANAGER_CPU_LIMIT_FACTOR);
+checkArgument(
+limitFactor >= 1,
+"%s should be greater or equal to 1.",
+KubernetesConfigOptions.TASK_MANAGER_CPU_LIMIT_FACTOR.key());
+return limitFactor;
+}
+
+public double getTaskManagerMemoryLimitFactor() {
+double limitFactor =
Review comment:
nit: limitFactor could be final.
##
File path:
flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClientTest.java
##
@@ -116,6 +117,8 @@ protected void setupFlinkConfig() {
flinkConfig.set(JobManagerOptions.PORT, RPC_PORT);
flinkConfig.set(BlobServerOptions.PORT,
Integer.toString(BLOB_SERVER_PORT));
flinkConfig.set(KubernetesConfigOptions.JOB_MANAGER_CPU,
JOB_MANAGER_CPU);
+flinkConfig.set(
Review comment:
I think we could leave the `KubernetesConfigOptions.JOB_MANAGER_CPU` as
default since we do not verify the resource limit in the tests of
`Fabric8FlinkKubeClientTest`.
--
This is an automated message from the Apache Git Service.
To respond to the me
[GitHub] [flink] zhuzhurk commented on a change in pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
zhuzhurk commented on a change in pull request #18102:
URL: https://github.com/apache/flink/pull/18102#discussion_r786442228
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopology.java
##
@@ -159,50 +162,79 @@ public static DefaultExecutionTopology fromExecutionGraph(
topologicallySortedJobVertices)
.getAllPipelinedRegions();
-ExecutionGraphIndex executionGraphIndex =
-computeExecutionGraphIndex(
-executionGraph.getAllExecutionVertices(),
-logicalPipelinedRegions,
-edgeManager);
-
-IndexedPipelinedRegions indexedPipelinedRegions =
-computePipelinedRegions(
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion.keySet(),
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion::get,
-executionGraphIndex.executionVerticesById::get,
-executionGraphIndex.resultPartitionsById::get);
-
-ensureCoLocatedVerticesInSameRegion(
-indexedPipelinedRegions.pipelinedRegions, executionGraph);
-
-return new DefaultExecutionTopology(
-executionGraphIndex.executionVerticesById,
-executionGraphIndex.executionVerticesList,
-executionGraphIndex.resultPartitionsById,
-indexedPipelinedRegions.pipelinedRegionsByVertex,
-indexedPipelinedRegions.pipelinedRegions,
-edgeManager);
-}
-
-private static ExecutionGraphIndex computeExecutionGraphIndex(
-Iterable executionVertices,
-Iterable logicalPipelinedRegions,
-EdgeManager edgeManager) {
-Map executionVerticesById =
new HashMap<>();
-List executionVerticesList = new ArrayList<>();
-Map
resultPartitionsById =
-new HashMap<>();
-Map>
-sortedExecutionVerticesInPipelinedRegion = new
IdentityHashMap<>();
-
-Map
logicalPipelinedRegionByJobVertexId =
+Map
logicalPipelinedRegionsByJobVertexId =
new HashMap<>();
for (DefaultLogicalPipelinedRegion logicalPipelinedRegion :
logicalPipelinedRegions) {
for (LogicalVertex vertex : logicalPipelinedRegion.getVertices()) {
-logicalPipelinedRegionByJobVertexId.put(vertex.getId(),
logicalPipelinedRegion);
+logicalPipelinedRegionsByJobVertexId.put(vertex.getId(),
logicalPipelinedRegion);
}
}
+return logicalPipelinedRegionsByJobVertexId;
+}
+
+public void notifyExecutionGraphUpdated(
+final DefaultExecutionGraph executionGraph,
+final List newJobVertices) {
+
+checkNotNull(executionGraph, "execution graph can not be null");
+
+final Set newVertexSet =
Review comment:
NIT: newVertexSet -> newJobVertexIds
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopology.java
##
@@ -159,50 +162,79 @@ public static DefaultExecutionTopology fromExecutionGraph(
topologicallySortedJobVertices)
.getAllPipelinedRegions();
-ExecutionGraphIndex executionGraphIndex =
-computeExecutionGraphIndex(
-executionGraph.getAllExecutionVertices(),
-logicalPipelinedRegions,
-edgeManager);
-
-IndexedPipelinedRegions indexedPipelinedRegions =
-computePipelinedRegions(
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion.keySet(),
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion::get,
-executionGraphIndex.executionVerticesById::get,
-executionGraphIndex.resultPartitionsById::get);
-
-ensureCoLocatedVerticesInSameRegion(
-indexedPipelinedRegions.pipelinedRegions, executionGraph);
-
-return new DefaultExecutionTopology(
-executionGraphIndex.executionVerticesById,
-executionGraphIndex.executionVerticesList,
-executionGraphIndex.resultPartitionsById,
-indexedPipelinedRegions.pipelinedRegionsByVertex,
-indexedPipelinedRegions.pipelinedRegions,
-edgeManager);
-}
-
-private static ExecutionGraphIndex computeExecutionGraphIndex(
-Iterable executionVertices,
-Iterable logicalPipelinedRegions,
-EdgeManager edgeManager) {
-Map executionVerticesById =
new HashMap<>();
-List executionVerticesList = new ArrayList<>();
-Map
resultPartitionsById =
-new HashMap<>(
[GitHub] [flink] flinkbot edited a comment on pull request #18303: [FLINK-25085][runtime] Add a scheduled thread pool for periodic tasks in RpcEndpoint
flinkbot edited a comment on pull request #18303: URL: https://github.com/apache/flink/pull/18303#issuecomment-1008005239 ## CI report: * 0e27f71b22e902f655820375325456a28ce378ec Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29601) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17098: [FLINK-15648][flink-kubernetes] Support to configure limit for CPU & memory
flinkbot edited a comment on pull request #17098: URL: https://github.com/apache/flink/pull/17098#issuecomment-910306059 ## CI report: * d3683df7f1e89088d8d6c255c92f6be38ff65fb3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23618) * 70fca6164f1fa6d430fe74168f70dcc754ea70c9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17814: [FLINK-24899][runtime] Enable data compression for blocking shuffle by default
flinkbot edited a comment on pull request #17814: URL: https://github.com/apache/flink/pull/17814#issuecomment-971129699 ## CI report: * 96cd76d4486a8096148b9c87d7f4cee71f54c17b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29602) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wsry closed pull request #17814: [FLINK-24899][runtime] Enable data compression for blocking shuffle by default
wsry closed pull request #17814: URL: https://github.com/apache/flink/pull/17814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24899) Enable data compression for blocking shuffle by default
[ https://issues.apache.org/jira/browse/FLINK-24899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-24899. --- Resolution: Fixed > Enable data compression for blocking shuffle by default > --- > > Key: FLINK-24899 > URL: https://issues.apache.org/jira/browse/FLINK-24899 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Yingjie Cao >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Currently, shuffle data compression is not enabled by default. Shuffle data > compression is important for blocking data shuffle and enabling shuffle data > compression by default can improve the usability. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24899) Enable data compression for blocking shuffle by default
[ https://issues.apache.org/jira/browse/FLINK-24899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477616#comment-17477616 ] Yingjie Cao commented on FLINK-24899: - Merged into master via a14c55482f60b73d3894a86a81c2ac6725acbb00. > Enable data compression for blocking shuffle by default > --- > > Key: FLINK-24899 > URL: https://issues.apache.org/jira/browse/FLINK-24899 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Yingjie Cao >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Currently, shuffle data compression is not enabled by default. Shuffle data > compression is important for blocking data shuffle and enabling shuffle data > compression by default can improve the usability. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17098: [FLINK-15648][flink-kubernetes] Support to configure limit for CPU & memory
flinkbot edited a comment on pull request #17098: URL: https://github.com/apache/flink/pull/17098#issuecomment-910306059 ## CI report: * d3683df7f1e89088d8d6c255c92f6be38ff65fb3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23618) * 70fca6164f1fa6d430fe74168f70dcc754ea70c9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29615) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29617) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25687) DropDelete is incorrect in CompactManager when outputLevel is zero
Jingsong Lee created FLINK-25687: Summary: DropDelete is incorrect in CompactManager when outputLevel is zero Key: FLINK-25687 URL: https://issues.apache.org/jira/browse/FLINK-25687 Project: Flink Issue Type: Bug Components: Table Store Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: table-store-0.1.0 When output level is zero, there may be have other files in level 0, we can not drop delete. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) * 6729dba080503a3f34f6441355d5c25091f8b375 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi opened a new pull request #11: [FLINK-25687] DropDelete is incorrect in CompactManager when outputLevel is zero
JingsongLi opened a new pull request #11: URL: https://github.com/apache/flink-table-store/pull/11 When output level is zero, there may be have other files in level 0, we can not drop delete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25687) DropDelete is incorrect in CompactManager when outputLevel is zero
[ https://issues.apache.org/jira/browse/FLINK-25687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25687: --- Labels: pull-request-available (was: ) > DropDelete is incorrect in CompactManager when outputLevel is zero > -- > > Key: FLINK-25687 > URL: https://issues.apache.org/jira/browse/FLINK-25687 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.1.0 > > > When output level is zero, there may be have other files in level 0, we can > not drop delete. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) * 6729dba080503a3f34f6441355d5c25091f8b375 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29618) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
flinkbot edited a comment on pull request #18350: URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932 ## CI report: * 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523) * 6c413ec28bb033a2dee3c06a60279971da1f324a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a change in pull request #11: [FLINK-25687] DropDelete is incorrect in CompactManager when outputLevel is zero
tsreaper commented on a change in pull request #11:
URL: https://github.com/apache/flink-table-store/pull/11#discussion_r786480063
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/mergetree/compact/CompactManager.java
##
@@ -75,7 +75,8 @@ public void submitCompaction(Levels levels) {
return;
}
boolean dropDelete =
-unit.outputLevel() >=
levels.nonEmptyHighestLevel();
+unit.outputLevel() != 0
+&& unit.outputLevel() >=
levels.nonEmptyHighestLevel();
Review comment:
add comments to explain the logic (for example level 0 is special
because key ranges from different files may overlap, and that compaction are
based on runs instead of files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18349: [FLINK-25609][table] Anonymous/inline tables don't require ObjectIdentifier anymore
flinkbot edited a comment on pull request #18349: URL: https://github.com/apache/flink/pull/18349#issuecomment-1011991081 ## CI report: * 0aa7220300adc4f5f9ef0aa59608f299b3b9ac29 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29584) * 8dd59b65de6e71d57da22d35686b0758ff3aed24 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
flinkbot edited a comment on pull request #18350: URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932 ## CI report: * 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523) * 6c413ec28bb033a2dee3c06a60279971da1f324a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29619) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul reassigned FLINK-25686: --- Assignee: Yufei Zhang > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18349: [FLINK-25609][table] Anonymous/inline tables don't require ObjectIdentifier anymore
flinkbot edited a comment on pull request #18349: URL: https://github.com/apache/flink/pull/18349#issuecomment-1011991081 ## CI report: * 0aa7220300adc4f5f9ef0aa59608f299b3b9ac29 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29584) * 8dd59b65de6e71d57da22d35686b0758ff3aed24 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29620) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477633#comment-17477633 ] Martijn Visser commented on FLINK-25686: [~affe] [~syhily] Will you pick up this ticket before the Flink 1.15 release branch is cut (on February 6th)? > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-25686: --- Issue Type: Technical Debt (was: New Feature) > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686 ] Martijn Visser deleted comment on FLINK-25686: was (Author: martijnvisser): [~affe] [~syhily] Will you pick up this ticket before the Flink 1.15 release branch is cut (on February 6th)? > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24246) Bump Pulsar to 2.9.0
[ https://issues.apache.org/jira/browse/FLINK-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477634#comment-17477634 ] Martijn Visser commented on FLINK-24246: [~affe] [~syhily] Will you pick up this ticket before the Flink 1.15 release branch is cut (on February 6th)? > Bump Pulsar to 2.9.0 > > > Key: FLINK-24246 > URL: https://issues.apache.org/jira/browse/FLINK-24246 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: 1.14.0, 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0 > > > Pulsar 2.8.1 has been released, the hack for getting TxnId from Pulsar > Transaction could be removed after bump flink-connector-pulsar's > pulsar-client-all to 2.8.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24668) Pulsar consumption data repeated output
[ https://issues.apache.org/jira/browse/FLINK-24668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-24668. -- Resolution: Cannot Reproduce [~余嘉悦] Since we don't have the necessary information to reproduce, I've closed this ticket. Please re-open if you still are experiencing the issues with information to reproduce. > Pulsar consumption data repeated output > --- > > Key: FLINK-24668 > URL: https://issues.apache.org/jira/browse/FLINK-24668 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.14.0 >Reporter: jiayue.yu >Assignee: Yufan Sheng >Priority: Major > > Flink-connector-pulsar uses the RecordsBySplits of Flink-connector-base to > process data, But recordsInCurrentSplit and splitsIterator are not empty when > nextSplit splitSiterator.hasNext () = false. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-20733) Introduce the upsert-pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-20733: --- Component/s: Connectors / Pulsar (was: Connectors / Common) > Introduce the upsert-pulsar Connector > - > > Key: FLINK-20733 > URL: https://issues.apache.org/jira/browse/FLINK-20733 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.13.0 >Reporter: Jianyun Zhao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) * 6729dba080503a3f34f6441355d5c25091f8b375 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29618) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
