[GitHub] [flink] yxu-valleytider commented on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
yxu-valleytider commented on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-532995742 PTAL @kl0u This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9714: [FLINK-14123]Change taskmanager.memory.fraction default value to 0.6
flinkbot commented on issue #9714: [FLINK-14123]Change taskmanager.memory.fraction default value to 0.6 URL: https://github.com/apache/flink/pull/9714#issuecomment-532994026 ## CI report: * 9c2f79d83a667cefc8364c4244f183e3798d9602 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks
flinkbot edited a comment on issue #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks URL: https://github.com/apache/flink/pull/9713#issuecomment-532988933 ## CI report: * f0fb6004eb0d0cf294f530aa0348ce23c5e701d8 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128286311) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster
flinkbot edited a comment on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster URL: https://github.com/apache/flink/pull/9711#issuecomment-532971678 ## CI report: * 2011905b5abe4cb332a60bc3f70378c777482924 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128281677) * 8f3ec4639c0c16591302cdd2a5b294d357903a22 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128282747) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuyang1706 commented on a change in pull request #9184: [FLINK-13339][ml] Add an implementation of pipeline's api
xuyang1706 commented on a change in pull request #9184: [FLINK-13339][ml] Add
an implementation of pipeline's api
URL: https://github.com/apache/flink/pull/9184#discussion_r326013357
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/batchoperator/BatchOperator.java
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.batchoperator;
+
+import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.batchoperator.source.TableSourceBatchOp;
+import org.apache.flink.ml.common.AlgoOperator;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class of batch algorithm operators.
+ */
+public abstract class BatchOperator> extends
AlgoOperator {
Review comment:
Thanks, provided.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuyang1706 commented on a change in pull request #9184: [FLINK-13339][ml] Add an implementation of pipeline's api
xuyang1706 commented on a change in pull request #9184: [FLINK-13339][ml] Add
an implementation of pipeline's api
URL: https://github.com/apache/flink/pull/9184#discussion_r326013260
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/batchoperator/BatchOperator.java
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.batchoperator;
+
+import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.batchoperator.source.TableSourceBatchOp;
+import org.apache.flink.ml.common.AlgoOperator;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class of batch algorithm operators.
+ */
+public abstract class BatchOperator> extends
AlgoOperator {
+
+ public BatchOperator() {
+ super();
+ }
+
+ public BatchOperator(Params params) {
+ super(params);
+ }
+
+ public > B link(B next) {
+ return linkTo(next);
+ }
+
+ public > B linkTo(B next) {
+ next.linkFrom(this);
+ return next;
+ }
+
+ public abstract T linkFrom(BatchOperator... inputs);
Review comment:
Thanks, added intro and example.
For there is no answer area of the above question, I put the answer here:
PipelineStage only supports single input and single output, it is the basic
unit for pipeline. AlgoOperator supports multi-input and multi-output. We’d
like to implement the algorithm with AlgoOperator, and PipelineStage’s fit and
transform function can call the AlgoOperator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9714: [FLINK-14123]Change taskmanager.memory.fraction default value to 0.6
flinkbot commented on issue #9714: [FLINK-14123]Change taskmanager.memory.fraction default value to 0.6 URL: https://github.com/apache/flink/pull/9714#issuecomment-532990781 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9c2f79d83a667cefc8364c4244f183e3798d9602 (Thu Sep 19 06:42:20 UTC 2019) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14123).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] liupc opened a new pull request #9714: [FLINK-14123]Change taskmanager.memory.fraction default value to 0.6
liupc opened a new pull request #9714: [FLINK-14123]Change taskmanager.memory.fraction default value to 0.6 URL: https://github.com/apache/flink/pull/9714 ## What is the purpose of the change The default JVM Option value `NewRatio` is 2, that means the old generation ocuppies 2/3 (0.66) of the heap memory, but we allocate 0.7 (`taskmanager.memory.fraction` default value) of the heap memory in `UnilateralSortMerger`, that's a little bit aggressive. This PR will change the default value to 0.6 to be more safe and stable. ## Brief change log - Changed the default value in TaskManagerOptions and docs and related scripts. ## Verifying this change This change is already covered by existing tests, and also tested with custom terasort application. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: ( no ) - The runtime per-record code paths (performance sensitive): (no ) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no ) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14123) Change taskmanager.memory.fraction default value to 0.6
[
https://issues.apache.org/jira/browse/FLINK-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-14123:
---
Labels: pull-request-available (was: )
> Change taskmanager.memory.fraction default value to 0.6
> ---
>
> Key: FLINK-14123
> URL: https://issues.apache.org/jira/browse/FLINK-14123
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Affects Versions: 1.9.0
>Reporter: liupengcheng
>Priority: Major
> Labels: pull-request-available
>
> Currently, we are testing flink batch task, such as terasort, however, it
> started only awhile then it failed due to OOM.
>
> {code:java}
> org.apache.flink.client.program.ProgramInvocationException: Job failed.
> (JobID: a807e1d635bd4471ceea4282477f8850)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:539)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort$.main(FlinkTeraSort.scala:89)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort.main(FlinkTeraSort.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:604)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:466)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1007)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1080)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1080)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259)
> ... 23 more
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
> Thread 'SortMerger Reading Thread' terminated due to an exception: GC
> overhead limit exceeded
> at
> org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650)
> at
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1109)
> at org.apache.flink.runtime.operators.NoOpDriver.run(NoOpDriver.java:82)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated
> due to an exception: GC overhead limit exceeded
> at
> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831)
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer.deserialize(BytePrimitiveArraySerializer.java:84)
> at
> org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer.deserialize(BytePrimitiveArraySerializer.java:33)
> at
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.deserialize(CaseClassSeriali
[GitHub] [flink] flinkbot commented on issue #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks
flinkbot commented on issue #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks URL: https://github.com/apache/flink/pull/9713#issuecomment-532988933 ## CI report: * f0fb6004eb0d0cf294f530aa0348ce23c5e701d8 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0
flinkbot edited a comment on issue #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0 URL: https://github.com/apache/flink/pull/9712#issuecomment-532984287 ## CI report: * f2b7710f65f478342de389c8e099799287ddf3f9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128285030) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-526715445 ## CI report: * dc9cdc1364b9050c3d2f3cba062310ff68d83b8e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125277359) * 5893967f343a3d2e3b4c92122303e00b01b0f8ff : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125282063) * ca8f65756da270da937077a2553a9506066b36bc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125303705) * fc346331b4e7460c75c2e4eb0eefd5f1d1bd7dc9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125561291) * 3156f2214291b0b18807c50675f7ac6b15e42ba4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125817723) * 66ba043a392e7be15c7aed2f7e2edcaef1472c15 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125818539) * df82b55379806ac8502ef92999a1abc7f9a0056b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/126191145) * 9fb1665010c88288eb49c5a9c6d1d2b81e257d71 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128271616) * 9abb55b018585e11703afe9f2494ae95fb129c0a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128277184) * 0301c85818f5cd92b694dbee070cd1a99895f495 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128285009) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-14123) Change taskmanager.memory.fraction default value to 0.6
liupengcheng created FLINK-14123:
Summary: Change taskmanager.memory.fraction default value to 0.6
Key: FLINK-14123
URL: https://issues.apache.org/jira/browse/FLINK-14123
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration
Affects Versions: 1.9.0
Reporter: liupengcheng
Currently, we are testing flink batch task, such as terasort, however, it
started only awhile then it failed due to OOM.
{code:java}
org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID:
a807e1d635bd4471ceea4282477f8850)
at
org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262)
at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
at
org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:539)
at
com.github.ehiggs.spark.terasort.FlinkTeraSort$.main(FlinkTeraSort.scala:89)
at
com.github.ehiggs.spark.terasort.FlinkTeraSort.main(FlinkTeraSort.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:604)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:466)
at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
at
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1007)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1080)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1080)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution
failed.
at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
at
org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259)
... 23 more
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread
'SortMerger Reading Thread' terminated due to an exception: GC overhead limit
exceeded
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650)
at
org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1109)
at org.apache.flink.runtime.operators.NoOpDriver.run(NoOpDriver.java:82)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated
due to an exception: GC overhead limit exceeded
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at
org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer.deserialize(BytePrimitiveArraySerializer.java:84)
at
org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer.deserialize(BytePrimitiveArraySerializer.java:33)
at
org.apache.flink.api.scala.typeutils.CaseClassSerializer.deserialize(CaseClassSerializer.scala:121)
at
org.apache.flink.api.scala.typeutils.CaseClassSerializer.deserialize(CaseClassSerializer.scala:114)
at
org.apache.flink.api.scala.typeutils.CaseClassSerializer.deserialize(CaseClassSerializer.scala:32)
at
org.apache.flink.runtime.plugable.ReusingDeserializationDelegate.read(ReusingDeserializationDelegate.java:57)
at
org.ap
[jira] [Comment Edited] (FLINK-13682) Translate "Code Style - Scala Guide" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933091#comment-16933091 ] Jeff Yang edited comment on FLINK-13682 at 9/19/19 6:23 AM: Hi, [~jark] I am already done ,Please take a look [PR|https://github.com/apache/flink-web/pull/267],Thanks . was (Author: highfei2...@126.com): Hi, [~jark] I am already done ,Please take a look this [PR|[https://github.com/apache/flink-web/pull/267]],Thanks . > Translate "Code Style - Scala Guide" page into Chinese > -- > > Key: FLINK-13682 > URL: https://issues.apache.org/jira/browse/FLINK-13682 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Project Website >Reporter: Jark Wu >Assignee: Jeff Yang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Translate page > https://flink.apache.org/zh/contributing/code-style-and-quality-scala.html > into Chinese. The page is located in > https://github.com/apache/flink-web/blob/asf-site/contributing/code-style-and-quality-scala.zh.md. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks
flinkbot commented on issue #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks URL: https://github.com/apache/flink/pull/9713#issuecomment-532985575 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f0fb6004eb0d0cf294f530aa0348ce23c5e701d8 (Thu Sep 19 06:23:41 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13682) Translate "Code Style - Scala Guide" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933091#comment-16933091 ] Jeff Yang commented on FLINK-13682: --- Hi, [~jark] I am already done ,Please take a look this [PR|[https://github.com/apache/flink-web/pull/267]],Thanks . > Translate "Code Style - Scala Guide" page into Chinese > -- > > Key: FLINK-13682 > URL: https://issues.apache.org/jira/browse/FLINK-13682 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Project Website >Reporter: Jark Wu >Assignee: Jeff Yang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Translate page > https://flink.apache.org/zh/contributing/code-style-and-quality-scala.html > into Chinese. The page is located in > https://github.com/apache/flink-web/blob/asf-site/contributing/code-style-and-quality-scala.zh.md. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] link3280 opened a new pull request #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks
link3280 opened a new pull request #9713: [hotfix][API/DataStream] Add missing assert for checkpoint lock checks URL: https://github.com/apache/flink/pull/9713 ## What is the purpose of the change The return value of `Thread.holdsLock(checkpointLock)` is unused, which are supposed to be validated by an assertion instead. ## Brief change log - Add assert clause to validate the return value of `Thread.holdsLock(checkpointLock)`. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: ( no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13682) Translate "Code Style - Scala Guide" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13682: --- Labels: pull-request-available (was: ) > Translate "Code Style - Scala Guide" page into Chinese > -- > > Key: FLINK-13682 > URL: https://issues.apache.org/jira/browse/FLINK-13682 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Project Website >Reporter: Jark Wu >Assignee: Jeff Yang >Priority: Major > Labels: pull-request-available > > Translate page > https://flink.apache.org/zh/contributing/code-style-and-quality-scala.html > into Chinese. The page is located in > https://github.com/apache/flink-web/blob/asf-site/contributing/code-style-and-quality-scala.zh.md. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0
flinkbot commented on issue #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0 URL: https://github.com/apache/flink/pull/9712#issuecomment-532984287 ## CI report: * f2b7710f65f478342de389c8e099799287ddf3f9 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/…
flinkbot edited a comment on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/… URL: https://github.com/apache/flink/pull/9683#issuecomment-531697778 ## CI report: * ba87beba470b132b99ed606411ffc43847a72d8f : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127766962) * 9838e772ccac5488aee7fb52837f21bcce94d882 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127936152) * 89765aad20b64adb2c240f818af45037f6e36e0d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128133420) * 421a64587f14f5116f6375f7b285bc6846bdc6ff : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128283861) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-526715445 ## CI report: * dc9cdc1364b9050c3d2f3cba062310ff68d83b8e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125277359) * 5893967f343a3d2e3b4c92122303e00b01b0f8ff : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125282063) * ca8f65756da270da937077a2553a9506066b36bc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125303705) * fc346331b4e7460c75c2e4eb0eefd5f1d1bd7dc9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125561291) * 3156f2214291b0b18807c50675f7ac6b15e42ba4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125817723) * 66ba043a392e7be15c7aed2f7e2edcaef1472c15 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125818539) * df82b55379806ac8502ef92999a1abc7f9a0056b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/126191145) * 9fb1665010c88288eb49c5a9c6d1d2b81e257d71 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128271616) * 9abb55b018585e11703afe9f2494ae95fb129c0a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128277184) * 0301c85818f5cd92b694dbee070cd1a99895f495 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] yxu-valleytider commented on a change in pull request #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileS
yxu-valleytider commented on a change in pull request #9581:
[FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to
allow for easier subclassing of StreamingFileSink
URL: https://github.com/apache/flink/pull/9581#discussion_r326002753
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java
##
@@ -162,47 +162,54 @@ protected StreamingFileSink(
* @return The builder where the remaining of the configuration
parameters for the sink can be configured.
* In order to instantiate the sink, call {@link
RowFormatBuilder#build()} after specifying the desired parameters.
*/
- public static StreamingFileSink.BulkFormatBuilder
forBulkFormat(
+ public static StreamingFileSink.BulkFormatBuilder> forBulkFormat(
final Path basePath, final BulkWriter.Factory
writerFactory) {
return new StreamingFileSink.BulkFormatBuilder<>(basePath,
writerFactory, new DateTimeBucketAssigner<>());
}
/**
* The base abstract class for the {@link RowFormatBuilder} and {@link
BulkFormatBuilder}.
*/
- protected abstract static class BucketsBuilder implements
Serializable {
+ protected abstract static class BucketsBuilder> implements Serializable {
Review comment:
DONE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] yxu-valleytider commented on a change in pull request #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileS
yxu-valleytider commented on a change in pull request #9581:
[FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to
allow for easier subclassing of StreamingFileSink
URL: https://github.com/apache/flink/pull/9581#discussion_r326000120
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java
##
@@ -264,37 +288,32 @@ private RowFormatBuilder(
subtaskIndex,
new PartFileConfig(partFilePrefix,
partFileSuffix));
}
-
- @VisibleForTesting
- StreamingFileSink.RowFormatBuilder
withBucketFactory(final BucketFactory factory) {
- return new RowFormatBuilder<>(basePath, encoder,
bucketAssigner, rollingPolicy, bucketCheckInterval,
Preconditions.checkNotNull(factory), partFilePrefix, partFileSuffix);
- }
}
/**
* A builder for configuring the sink for bulk-encoding formats, e.g.
Parquet/ORC.
*/
@PublicEvolving
- public static class BulkFormatBuilder extends
StreamingFileSink.BucketsBuilder {
+ public static class BulkFormatBuilder> extends
StreamingFileSink.BucketsBuilder {
Review comment:
DONE similar changes as `RowFormatBuilder`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-14037) Deserializing the input/output formats failed: unread block data
[
https://issues.apache.org/jira/browse/FLINK-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933087#comment-16933087
]
liupengcheng commented on FLINK-14037:
--
[~zhuzh]
Yes, that's weird, Maybe I think your test is not complete, you can change
x.y.z.B to a.b.c.B, and test it again to check whether the results are still
the same. I'm not sure about the behavior of the defineClass, but I think the
delegation loading only happen in `loadClass` method, in defineClass, it will
load with current loader. Maybe I'm wrong, but if we cannot explain my case,
then I'm not convinced that `alwaysParentFirstLoaderPatterns` will always make
class in the list loaded by parent loader unless it does not exist in the
parent classloader. .
> Deserializing the input/output formats failed: unread block data
>
>
> Key: FLINK-14037
> URL: https://issues.apache.org/jira/browse/FLINK-14037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0
> Environment: flink 1.9.0
> app jar use `flink-shaded-hadoop-2` dependencies to avoid some confilicts
>
>Reporter: liupengcheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recently, we encountered the following issue when testing flink 1.9.0:
> {code:java}
> org.apache.flink.client.program.ProgramInvocationException: Could not
> retrieve the execution result. (JobID: 8ffbc071dda81d6f8005c02be8adde6b)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:255)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:539)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort$.main(FlinkTeraSort.scala:89)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort.main(FlinkTeraSort.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
> submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$8(RestClusterClient.java:382)
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:263)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:5
[GitHub] [flink] yxu-valleytider commented on a change in pull request #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileS
yxu-valleytider commented on a change in pull request #9581:
[FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to
allow for easier subclassing of StreamingFileSink
URL: https://github.com/apache/flink/pull/9581#discussion_r325999744
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java
##
@@ -224,35 +231,52 @@ private RowFormatBuilder(
this.partFileSuffix =
Preconditions.checkNotNull(partFileSuffix);
}
- public StreamingFileSink.RowFormatBuilder
withBucketCheckInterval(final long interval) {
- return new RowFormatBuilder<>(basePath, encoder,
bucketAssigner, rollingPolicy, interval, bucketFactory, partFilePrefix,
partFileSuffix);
+ public T withBucketCheckInterval(final long interval) {
+ this.bucketCheckInterval = interval;
+ return self();
}
- public StreamingFileSink.RowFormatBuilder
withBucketAssigner(final BucketAssigner assigner) {
- return new RowFormatBuilder<>(basePath, encoder,
Preconditions.checkNotNull(assigner), rollingPolicy, bucketCheckInterval,
bucketFactory, partFilePrefix, partFileSuffix);
+ public T withBucketAssigner(final BucketAssigner
assigner) {
+ this.bucketAssigner =
Preconditions.checkNotNull(assigner);
+ return self();
}
- public StreamingFileSink.RowFormatBuilder
withRollingPolicy(final RollingPolicy policy) {
- return new RowFormatBuilder<>(basePath, encoder,
bucketAssigner, Preconditions.checkNotNull(policy), bucketCheckInterval,
bucketFactory, partFilePrefix, partFileSuffix);
+ public T withRollingPolicy(final RollingPolicy
policy) {
+ this.rollingPolicy = Preconditions.checkNotNull(policy);
+ return self();
}
- public StreamingFileSink.RowFormatBuilder
withBucketAssignerAndPolicy(final BucketAssigner assigner, final
RollingPolicy policy) {
- return new RowFormatBuilder<>(basePath, encoder,
Preconditions.checkNotNull(assigner), Preconditions.checkNotNull(policy),
bucketCheckInterval, new DefaultBucketFactoryImpl<>(), partFilePrefix,
partFileSuffix);
+ public T withBucketAssignerAndPolicy(final BucketAssigner assigner, final RollingPolicy policy) {
+ this.bucketAssigner =
Preconditions.checkNotNull(assigner);
+ this.rollingPolicy = Preconditions.checkNotNull(policy);
+ return self();
}
- public StreamingFileSink.RowFormatBuilder
withPartFilePrefix(final String partPrefix) {
- return new RowFormatBuilder<>(basePath, encoder,
bucketAssigner, rollingPolicy, bucketCheckInterval, bucketFactory, partPrefix,
partFileSuffix);
+ public T withPartFilePrefix(final String partPrefix) {
+ this.partFilePrefix = partPrefix;
+ return self();
}
- public StreamingFileSink.RowFormatBuilder
withPartFileSuffix(final String partSuffix) {
- return new RowFormatBuilder<>(basePath, encoder,
bucketAssigner, rollingPolicy, bucketCheckInterval, bucketFactory,
partFilePrefix, partSuffix);
+ public T withPartFileSuffix(final String partSuffix) {
+ this.partFileSuffix = partSuffix;
+ return self();
+ }
+
+ public StreamingFileSink.RowFormatBuilder>
newBuilderWithBucketAssignerAndPolicy(final BucketAssigner assigner,
final RollingPolicy policy) {
+ return new RowFormatBuilder<>(basePath, encoder,
Preconditions.checkNotNull(assigner), Preconditions.checkNotNull(policy),
bucketCheckInterval, new DefaultBucketFactoryImpl<>(), partFilePrefix,
partFileSuffix);
Review comment:
DONE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] yxu-valleytider commented on a change in pull request #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileS
yxu-valleytider commented on a change in pull request #9581:
[FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to
allow for easier subclassing of StreamingFileSink
URL: https://github.com/apache/flink/pull/9581#discussion_r326002736
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java
##
@@ -162,47 +162,54 @@ protected StreamingFileSink(
* @return The builder where the remaining of the configuration
parameters for the sink can be configured.
* In order to instantiate the sink, call {@link
RowFormatBuilder#build()} after specifying the desired parameters.
*/
- public static StreamingFileSink.BulkFormatBuilder
forBulkFormat(
+ public static StreamingFileSink.BulkFormatBuilder> forBulkFormat(
final Path basePath, final BulkWriter.Factory
writerFactory) {
return new StreamingFileSink.BulkFormatBuilder<>(basePath,
writerFactory, new DateTimeBucketAssigner<>());
}
/**
* The base abstract class for the {@link RowFormatBuilder} and {@link
BulkFormatBuilder}.
*/
- protected abstract static class BucketsBuilder implements
Serializable {
+ protected abstract static class BucketsBuilder> implements Serializable {
private static final long serialVersionUID = 1L;
+ protected static final long DEFAULT_BUCKET_CHECK_INTERVAL = 60L
* 1000L;
+
+ @SuppressWarnings("unchecked")
+ protected T self() {
+ return (T) this;
+ }
+
abstract Buckets createBuckets(final int
subtaskIndex) throws IOException;
}
/**
* A builder for configuring the sink for row-wise encoding formats.
*/
@PublicEvolving
- public static class RowFormatBuilder extends
StreamingFileSink.BucketsBuilder {
+ public static class RowFormatBuilder> extends StreamingFileSink.BucketsBuilder {
private static final long serialVersionUID = 1L;
- private final long bucketCheckInterval;
+ protected long bucketCheckInterval;
Review comment:
DONE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] yxu-valleytider commented on a change in pull request #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileS
yxu-valleytider commented on a change in pull request #9581:
[FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to
allow for easier subclassing of StreamingFileSink
URL: https://github.com/apache/flink/pull/9581#discussion_r326002700
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java
##
@@ -162,47 +162,54 @@ protected StreamingFileSink(
* @return The builder where the remaining of the configuration
parameters for the sink can be configured.
* In order to instantiate the sink, call {@link
RowFormatBuilder#build()} after specifying the desired parameters.
*/
- public static StreamingFileSink.BulkFormatBuilder
forBulkFormat(
+ public static StreamingFileSink.BulkFormatBuilder> forBulkFormat(
final Path basePath, final BulkWriter.Factory
writerFactory) {
return new StreamingFileSink.BulkFormatBuilder<>(basePath,
writerFactory, new DateTimeBucketAssigner<>());
}
/**
* The base abstract class for the {@link RowFormatBuilder} and {@link
BulkFormatBuilder}.
*/
- protected abstract static class BucketsBuilder implements
Serializable {
+ protected abstract static class BucketsBuilder> implements Serializable {
private static final long serialVersionUID = 1L;
+ protected static final long DEFAULT_BUCKET_CHECK_INTERVAL = 60L
* 1000L;
+
+ @SuppressWarnings("unchecked")
+ protected T self() {
+ return (T) this;
+ }
+
abstract Buckets createBuckets(final int
subtaskIndex) throws IOException;
}
/**
* A builder for configuring the sink for row-wise encoding formats.
*/
@PublicEvolving
- public static class RowFormatBuilder extends
StreamingFileSink.BucketsBuilder {
+ public static class RowFormatBuilder> extends StreamingFileSink.BucketsBuilder {
private static final long serialVersionUID = 1L;
- private final long bucketCheckInterval;
+ protected long bucketCheckInterval;
private final Path basePath;
- private final Encoder encoder;
+ private Encoder encoder;
- private final BucketAssigner bucketAssigner;
+ private BucketAssigner bucketAssigner;
- private final RollingPolicy rollingPolicy;
+ private RollingPolicy rollingPolicy;
- private final BucketFactory bucketFactory;
+ private BucketFactory bucketFactory;
- private final String partFilePrefix;
+ private String partFilePrefix;
- private final String partFileSuffix;
+ private String partFileSuffix;
- RowFormatBuilder(Path basePath, Encoder encoder,
BucketAssigner bucketAssigner) {
- this(basePath, encoder, bucketAssigner,
DefaultRollingPolicy.builder().build(), 60L * 1000L, new
DefaultBucketFactoryImpl<>(), PartFileConfig.DEFAULT_PART_PREFIX,
PartFileConfig.DEFAULT_PART_SUFFIX);
+ protected RowFormatBuilder(Path basePath, Encoder encoder,
BucketAssigner bucketAssigner) {
+ this(basePath, encoder, bucketAssigner,
DefaultRollingPolicy.builder().build(), DEFAULT_BUCKET_CHECK_INTERVAL, new
DefaultBucketFactoryImpl<>(), PartFileConfig.DEFAULT_PART_PREFIX,
PartFileConfig.DEFAULT_PART_SUFFIX);
}
private RowFormatBuilder(
Review comment:
DONE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster
flinkbot edited a comment on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster URL: https://github.com/apache/flink/pull/9711#issuecomment-532971678 ## CI report: * 2011905b5abe4cb332a60bc3f70378c777482924 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128281677) * 8f3ec4639c0c16591302cdd2a5b294d357903a22 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128282747) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/…
flinkbot edited a comment on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/… URL: https://github.com/apache/flink/pull/9683#issuecomment-531697778 ## CI report: * ba87beba470b132b99ed606411ffc43847a72d8f : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127766962) * 9838e772ccac5488aee7fb52837f21bcce94d882 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127936152) * 89765aad20b64adb2c240f818af45037f6e36e0d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128133420) * 421a64587f14f5116f6375f7b285bc6846bdc6ff : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0
flinkbot commented on issue #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0 URL: https://github.com/apache/flink/pull/9712#issuecomment-532978281 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f2b7710f65f478342de389c8e099799287ddf3f9 (Thu Sep 19 05:54:17 UTC 2019) **Warnings:** * **4 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13656).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13656) Upgrade Calcite dependency to 1.21
[ https://issues.apache.org/jira/browse/FLINK-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13656: --- Labels: pull-request-available (was: ) > Upgrade Calcite dependency to 1.21 > -- > > Key: FLINK-13656 > URL: https://issues.apache.org/jira/browse/FLINK-13656 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Legacy Planner, Table SQL / Planner >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > > Umbrella issue for all tasks related to the next Calcite upgrade to 1.21.x > release > Calcite 1.21 has been released recently, we need to upgrade to version 1.21 > for these reasons: > - Previously we have made some temp code to support full data types in sql > parser, since CALCITE-3213 has been resolved, we can do some refactoring for > these codes; > - We also fixed some important bug for Join which bring in from Calcite 1.20 > join like expression promotion, such as CALCITE-3170, CALCITE-3171. > - CALCITE-2302 has been resolved, there is possibility we support implicit > type coercion for Flink now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] liupc commented on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/…
liupc commented on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/… URL: https://github.com/apache/flink/pull/9683#issuecomment-532977900 @aljoscha @TisonKun I find it's hard to make the classloader outside the packagedProgram, there are lots of logic and deps in PackagedProgram to build the classpaths for the classsloader. So I just get the `classLoaderResoloverOrder` and `alwaysParentFirstLoaderPatterns` at client and pass them to the constructor arguments of the PackagedProgram. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] danny0405 opened a new pull request #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0
danny0405 opened a new pull request #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0 URL: https://github.com/apache/flink/pull/9712 ### SQL Parser Some refactoring in flink-sql-parser ### Flink Planner - `org.apache.calcite.rel.core.Match#partitionKeys` has changed from `List` to `ImmutableBitSet`, since CALCITE-1935, - `org.apache.calcite.rel.core.Match#copy` has been removed since CALCITE-1935 part1. - `org.apache.flink.table.api.stream.sql.JoinTest#testRightOuterJoinEquiAndLocalPred`, `#testLeftOuterJoinEquiAndLocalPred`, `org.apache.flink.table.api.batch.sql.JoinTest#testFullOuterJoinEquiAndLocalPred`, `#testRightOuterJoinEquiAndLocalPred`, `#testLeftOuterJoinEquiAndLocalPred` plan has changed, but it is still correct. since CALCITE-3101 - ConverterRule that use `RelOptRule#description` was affected because the default description generation rule has changed since CALCITE-3115. - `org.apache.flink.table.api.stream.table.AggregateTest#testMultiDistinctAggregateSameFieldOnHopWindow`, `org.apache.flink.table.api.stream.sql.DistinctAggregateTest#testMultiDistinctAggregateSameFieldOnHopWindow` max distinct was removed. Since CALCITE-3159 ### Blink Planner - The plan change mostly because of the same reason with flink planner - `SqlFunction` was copied from CALCITE because of CALCITE-3360 - org.apache.flink.table.planner.plan.batch.sql#testMinusWithNestedTypes IS NOT DISTINCT FROM is not expanded now since CALCITE-3174 - `FlinkTypeFactory#createDecimalQuotient` was moved to `FlinkTypeSystem#deriveDecimalDivideType `due to CALCITE-3187 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly.
flinkbot edited a comment on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly. URL: https://github.com/apache/flink/pull/9710#issuecomment-532961713 ## CI report: * 9df06c32a009c53dce3f13c7de12892eb8563213 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128278955) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster
flinkbot edited a comment on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster URL: https://github.com/apache/flink/pull/9711#issuecomment-532971678 ## CI report: * 2011905b5abe4cb332a60bc3f70378c777482924 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128281677) * 8f3ec4639c0c16591302cdd2a5b294d357903a22 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster
flinkbot commented on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster URL: https://github.com/apache/flink/pull/9711#issuecomment-532971678 ## CI report: * 2011905b5abe4cb332a60bc3f70378c777482924 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14033) Distributed caches are not registered in Yarn Per Job Cluster Mode
[ https://issues.apache.org/jira/browse/FLINK-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933057#comment-16933057 ] Zhenqiu Huang commented on FLINK-14033: --- [~trohrmann][~fly_in_gis] Would you please take a look at the fix? Thank you in advance. > Distributed caches are not registered in Yarn Per Job Cluster Mode > -- > > Key: FLINK-14033 > URL: https://issues.apache.org/jira/browse/FLINK-14033 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.0 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CacheFiles in StreamExecutionEnvironment is not used in Job Submission in the > Yarn per job cluster mode. Compare to the job submission in session cluster > mode that will upload distributed cache files onto http server in application > master, we should get the cache files in job graph and register them into > blob store in YarnJobClusterEntrypoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster
flinkbot commented on issue #9711: [FLINK-14033] upload user artifacts for yarn job cluster URL: https://github.com/apache/flink/pull/9711#issuecomment-532968944 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 2011905b5abe4cb332a60bc3f70378c777482924 (Thu Sep 19 05:13:41 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14033) Distributed caches are not registered in Yarn Per Job Cluster Mode
[ https://issues.apache.org/jira/browse/FLINK-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14033: --- Labels: pull-request-available (was: ) > Distributed caches are not registered in Yarn Per Job Cluster Mode > -- > > Key: FLINK-14033 > URL: https://issues.apache.org/jira/browse/FLINK-14033 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.0 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Minor > Labels: pull-request-available > > CacheFiles in StreamExecutionEnvironment is not used in Job Submission in the > Yarn per job cluster mode. Compare to the job submission in session cluster > mode that will upload distributed cache files onto http server in application > master, we should get the cache files in job graph and register them into > blob store in YarnJobClusterEntrypoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] HuangZhenQiu opened a new pull request #9711: [FLINK-14033] upload user artifacts for yarn job cluster
HuangZhenQiu opened a new pull request #9711: [FLINK-14033] upload user artifacts for yarn job cluster URL: https://github.com/apache/flink/pull/9711 ## What is the purpose of the change Upload user artifacts that registered in distributed caches for yarn job clusters. Currently, distributed caches are not handled correctly. The solution proposed is to upload local files onto the remote file system and update the registered file in cache entry. ## Brief change-log - Modified the AbstractClusterDescriptor to handle with artifacts for per job mode before starting application master. ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
carp84 commented on a change in pull request #9501: [FLINK-12697] [State
Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#discussion_r325990356
##
File path:
flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/CopyOnWriteSkipListStateMap.java
##
@@ -0,0 +1,1527 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.heap;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.ByteBufferUtils;
+import org.apache.flink.runtime.state.StateEntry;
+import org.apache.flink.runtime.state.StateTransformationFunction;
+import org.apache.flink.runtime.state.heap.space.Allocator;
+import org.apache.flink.runtime.state.heap.space.Chunk;
+import org.apache.flink.runtime.state.heap.space.SpaceUtils;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.ResourceGuard;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.Spliterators;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.flink.runtime.state.heap.SkipListUtils.HEAD_NODE;
+import static org.apache.flink.runtime.state.heap.SkipListUtils.NIL_NODE;
+import static
org.apache.flink.runtime.state.heap.SkipListUtils.NIL_VALUE_POINTER;
+
+/**
+ * Implementation of state map which is based on skip list with copy-on-write
support. states will
+ * be serialized to bytes and stored in the space allocated with the given
allocator.
+ *
+ * @param type of key
+ * @param type of namespace
+ * @param type of state
+ */
+public class CopyOnWriteSkipListStateMap extends StateMap
implements AutoCloseable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CopyOnWriteSkipListStateMap.class);
+
+ /**
+* Default max number of logically-removed keys to delete one time.
+*/
+ private static final int DEFAULT_MAX_KEYS_TO_DELETE_ONE_TIME = 3;
+
+ /**
+* Default ratio of the logically-removed keys to trigger deletion when
snapshot.
+*/
+ private static final float DEFAULT_LOGICAL_REMOVED_KEYS_RATIO = 0.2f;
+
+ /**
+* The serializer used to serialize the key and namespace to bytes
stored in skip list.
+*/
+ private final SkipListKeySerializer skipListKeySerializer;
+
+ /**
+* The serializer used to serialize the state to bytes stored in skip
list.
+*/
+ private final SkipListValueSerializer skipListValueSerializer;
+
+ /**
+* Space allocator.
+*/
+ private final Allocator spaceAllocator;
+
+ /**
+* The level index header.
+*/
+ private final LevelIndexHeader levelIndexHeader;
+
+ /**
+* Seed to generate random index level.
+*/
+ private int randomSeed;
+
+ /**
+* The current version of this map. Used for copy-on-write mechanics.
+*/
+ private int stateMapVersion;
+
+ /**
+* The highest version of this map that is still required by any
unreleased snapshot.
+*/
+ private int highestRequiredSnapshotVersion;
+
+ /**
+* Snapshots no more than this version must have been finished, but
there may be some
+* snapshots more than this version are still running.
+*/
+ private volatile int highestFinishedSnapshotVersion;
+
+ /**
+* Maintains an ordered set of version ids that are still us
[GitHub] [flink] flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-526715445 ## CI report: * dc9cdc1364b9050c3d2f3cba062310ff68d83b8e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125277359) * 5893967f343a3d2e3b4c92122303e00b01b0f8ff : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125282063) * ca8f65756da270da937077a2553a9506066b36bc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125303705) * fc346331b4e7460c75c2e4eb0eefd5f1d1bd7dc9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125561291) * 3156f2214291b0b18807c50675f7ac6b15e42ba4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125817723) * 66ba043a392e7be15c7aed2f7e2edcaef1472c15 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125818539) * df82b55379806ac8502ef92999a1abc7f9a0056b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/126191145) * 9fb1665010c88288eb49c5a9c6d1d2b81e257d71 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128271616) * 9abb55b018585e11703afe9f2494ae95fb129c0a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128277184) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly.
flinkbot edited a comment on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly. URL: https://github.com/apache/flink/pull/9710#issuecomment-532961713 ## CI report: * 9df06c32a009c53dce3f13c7de12892eb8563213 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128278955) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
carp84 commented on a change in pull request #9501: [FLINK-12697] [State
Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#discussion_r325987134
##
File path:
flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/CopyOnWriteSkipListStateMap.java
##
@@ -0,0 +1,1527 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.heap;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.ByteBufferUtils;
+import org.apache.flink.runtime.state.StateEntry;
+import org.apache.flink.runtime.state.StateTransformationFunction;
+import org.apache.flink.runtime.state.heap.space.Allocator;
+import org.apache.flink.runtime.state.heap.space.Chunk;
+import org.apache.flink.runtime.state.heap.space.SpaceUtils;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.ResourceGuard;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.Spliterators;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.flink.runtime.state.heap.SkipListUtils.HEAD_NODE;
+import static org.apache.flink.runtime.state.heap.SkipListUtils.NIL_NODE;
+import static
org.apache.flink.runtime.state.heap.SkipListUtils.NIL_VALUE_POINTER;
+
+/**
+ * Implementation of state map which is based on skip list with copy-on-write
support. states will
+ * be serialized to bytes and stored in the space allocated with the given
allocator.
+ *
+ * @param type of key
+ * @param type of namespace
+ * @param type of state
+ */
+public class CopyOnWriteSkipListStateMap extends StateMap
implements AutoCloseable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CopyOnWriteSkipListStateMap.class);
+
+ /**
+* Default max number of logically-removed keys to delete one time.
+*/
+ private static final int DEFAULT_MAX_KEYS_TO_DELETE_ONE_TIME = 3;
+
+ /**
+* Default ratio of the logically-removed keys to trigger deletion when
snapshot.
+*/
+ private static final float DEFAULT_LOGICAL_REMOVED_KEYS_RATIO = 0.2f;
+
+ /**
+* The serializer used to serialize the key and namespace to bytes
stored in skip list.
+*/
+ private final SkipListKeySerializer skipListKeySerializer;
+
+ /**
+* The serializer used to serialize the state to bytes stored in skip
list.
+*/
+ private final SkipListValueSerializer skipListValueSerializer;
+
+ /**
+* Space allocator.
+*/
+ private final Allocator spaceAllocator;
+
+ /**
+* The level index header.
+*/
+ private final LevelIndexHeader levelIndexHeader;
+
+ /**
+* Seed to generate random index level.
+*/
+ private int randomSeed;
+
+ /**
+* The current version of this map. Used for copy-on-write mechanics.
+*/
+ private int stateMapVersion;
+
+ /**
+* The highest version of this map that is still required by any
unreleased snapshot.
+*/
+ private int highestRequiredSnapshotVersion;
+
+ /**
+* Snapshots no more than this version must have been finished, but
there may be some
+* snapshots more than this version are still running.
+*/
+ private volatile int highestFinishedSnapshotVersion;
+
+ /**
+* Maintains an ordered set of version ids that are still us
[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
carp84 commented on a change in pull request #9501: [FLINK-12697] [State
Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#discussion_r325986596
##
File path:
flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/CopyOnWriteSkipListStateMap.java
##
@@ -0,0 +1,1527 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.heap;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.ByteBufferUtils;
+import org.apache.flink.runtime.state.StateEntry;
+import org.apache.flink.runtime.state.StateTransformationFunction;
+import org.apache.flink.runtime.state.heap.space.Allocator;
+import org.apache.flink.runtime.state.heap.space.Chunk;
+import org.apache.flink.runtime.state.heap.space.SpaceUtils;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.ResourceGuard;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.Spliterators;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.flink.runtime.state.heap.SkipListUtils.HEAD_NODE;
+import static org.apache.flink.runtime.state.heap.SkipListUtils.NIL_NODE;
+import static
org.apache.flink.runtime.state.heap.SkipListUtils.NIL_VALUE_POINTER;
+
+/**
+ * Implementation of state map which is based on skip list with copy-on-write
support. states will
+ * be serialized to bytes and stored in the space allocated with the given
allocator.
+ *
+ * @param type of key
+ * @param type of namespace
+ * @param type of state
+ */
+public class CopyOnWriteSkipListStateMap extends StateMap
implements AutoCloseable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CopyOnWriteSkipListStateMap.class);
+
+ /**
+* Default max number of logically-removed keys to delete one time.
+*/
+ private static final int DEFAULT_MAX_KEYS_TO_DELETE_ONE_TIME = 3;
+
+ /**
+* Default ratio of the logically-removed keys to trigger deletion when
snapshot.
+*/
+ private static final float DEFAULT_LOGICAL_REMOVED_KEYS_RATIO = 0.2f;
+
+ /**
+* The serializer used to serialize the key and namespace to bytes
stored in skip list.
+*/
+ private final SkipListKeySerializer skipListKeySerializer;
+
+ /**
+* The serializer used to serialize the state to bytes stored in skip
list.
+*/
+ private final SkipListValueSerializer skipListValueSerializer;
+
+ /**
+* Space allocator.
+*/
+ private final Allocator spaceAllocator;
+
+ /**
+* The level index header.
+*/
+ private final LevelIndexHeader levelIndexHeader;
+
+ /**
+* Seed to generate random index level.
+*/
+ private int randomSeed;
+
+ /**
+* The current version of this map. Used for copy-on-write mechanics.
+*/
+ private int stateMapVersion;
+
+ /**
+* The highest version of this map that is still required by any
unreleased snapshot.
+*/
+ private int highestRequiredSnapshotVersion;
+
+ /**
+* Snapshots no more than this version must have been finished, but
there may be some
+* snapshots more than this version are still running.
+*/
+ private volatile int highestFinishedSnapshotVersion;
+
+ /**
+* Maintains an ordered set of version ids that are still us
[GitHub] [flink] flinkbot commented on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly.
flinkbot commented on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly. URL: https://github.com/apache/flink/pull/9710#issuecomment-532961713 ## CI report: * 9df06c32a009c53dce3f13c7de12892eb8563213 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly.
flinkbot commented on issue #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly. URL: https://github.com/apache/flink/pull/9710#issuecomment-532959252 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9df06c32a009c53dce3f13c7de12892eb8563213 (Thu Sep 19 04:22:11 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11859) Improve SpanningRecordSerializer performance by serializing record length to serialization buffer directly
[ https://issues.apache.org/jira/browse/FLINK-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-11859: Fix Version/s: 1.10.0 > Improve SpanningRecordSerializer performance by serializing record length to > serialization buffer directly > -- > > Key: FLINK-11859 > URL: https://issues.apache.org/jira/browse/FLINK-11859 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Yingjie Cao >Assignee: Yingjie Cao >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > In the current implementation of SpanningRecordSerializer, the length of a > record is serialized to an intermediate length buffer and then copied to the > target buffer. Actually, the length filed can be serialized directly to the > data buffer (serializationBuffer), which can avoid the copy of length buffer. > Though the total bytes copied remain unchanged, it one copy of a small record > which incurs high overhead. The flink-benchmarks shows it can improve > performance and the test results are as follows. > Result with the optimization: > |Benchmark|Mode|Threads|Samples|Score|Score Error (99.9%)|Unit|Param: > channelsFlushTimeout|Param: stateBackend| > |KeyByBenchmarks.arrayKeyBy|thrpt|1|30|2228.049605|77.631804|ops/ms| | | > |KeyByBenchmarks.tupleKeyBy|thrpt|1|30|3968.361739|193.501755|ops/ms| | | > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|3030.016702|29.272713|ops/ms| > |MEMORY| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2754.77678|26.215395|ops/ms| > |FS| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|3001.957606|29.288019|ops/ms| > |FS_ASYNC| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|123.698984|3.339233|ops/ms| > |ROCKS| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|126.252137|1.137735|ops/ms| > |ROCKS_INC| > |SerializationFrameworkMiniBenchmarks.serializerAvro|thrpt|1|30|323.658098|5.855697|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerKryo|thrpt|1|30|183.34423|3.710787|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerPojo|thrpt|1|30|404.380233|5.131744|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerRow|thrpt|1|30|527.193369|10.176726|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerTuple|thrpt|1|30|550.073024|11.724412|ops/ms| > | | > |StreamNetworkBroadcastThroughputBenchmarkExecutor.networkBroadcastThroughput|thrpt|1|30|564.690627|13.766809|ops/ms| > | | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|49918.11806|2324.234776|ops/ms|100,100ms| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|10443.63491|315.835962|ops/ms|100,100ms,SSL| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|21387.47608|2779.832704|ops/ms|1000,1ms| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|26585.85453|860.243347|ops/ms|1000,100ms| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|8252.563405|947.129028|ops/ms|1000,100ms,SSL| > | > |SumLongsBenchmark.benchmarkCount|thrpt|1|30|8806.021402|263.995836|ops/ms| | > | > |WindowBenchmarks.globalWindow|thrpt|1|30|4573.620126|112.099391|ops/ms| | | > |WindowBenchmarks.sessionWindow|thrpt|1|30|585.246412|7.026569|ops/ms| | | > |WindowBenchmarks.slidingWindow|thrpt|1|30|449.302134|4.123669|ops/ms| | | > |WindowBenchmarks.tumblingWindow|thrpt|1|30|2979.806858|33.818909|ops/ms| | | > |StreamNetworkLatencyBenchmarkExecutor.networkLatency1to1|avgt|1|30|12.842865|0.13796|ms/op| > | | > Result without the optimization: > > |Benchmark|Mode|Threads|Samples|Score|Score Error (99.9%)|Unit|Param: > channelsFlushTimeout|Param: stateBackend| > |KeyByBenchmarks.arrayKeyBy|thrpt|1|30|2060.241715|59.898485|ops/ms| | | > |KeyByBenchmarks.tupleKeyBy|thrpt|1|30|3645.306819|223.821719|ops/ms| | | > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2992.698822|36.978115|ops/ms| > |MEMORY| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2756.10949|27.798937|ops/ms| > |FS| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2965.969876|44.159793|ops/ms| > |FS_ASYNC| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|125.506942|1.245978|ops/ms| > |ROCKS| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|127.258737|1.190588|ops/ms| > |ROCKS_INC| > |SerializationFrameworkMiniBenchmarks.serializerAvro|thrpt|1|30|316.497954|8.309241|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerKryo|thrpt|1|30|189.065149|6.302073|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerPojo|thrpt|1|30|391.5
[jira] [Updated] (FLINK-11859) Improve SpanningRecordSerializer performance by serializing record length to serialization buffer directly
[ https://issues.apache.org/jira/browse/FLINK-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11859: --- Labels: pull-request-available (was: ) > Improve SpanningRecordSerializer performance by serializing record length to > serialization buffer directly > -- > > Key: FLINK-11859 > URL: https://issues.apache.org/jira/browse/FLINK-11859 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Yingjie Cao >Assignee: Yingjie Cao >Priority: Major > Labels: pull-request-available > > In the current implementation of SpanningRecordSerializer, the length of a > record is serialized to an intermediate length buffer and then copied to the > target buffer. Actually, the length filed can be serialized directly to the > data buffer (serializationBuffer), which can avoid the copy of length buffer. > Though the total bytes copied remain unchanged, it one copy of a small record > which incurs high overhead. The flink-benchmarks shows it can improve > performance and the test results are as follows. > Result with the optimization: > |Benchmark|Mode|Threads|Samples|Score|Score Error (99.9%)|Unit|Param: > channelsFlushTimeout|Param: stateBackend| > |KeyByBenchmarks.arrayKeyBy|thrpt|1|30|2228.049605|77.631804|ops/ms| | | > |KeyByBenchmarks.tupleKeyBy|thrpt|1|30|3968.361739|193.501755|ops/ms| | | > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|3030.016702|29.272713|ops/ms| > |MEMORY| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2754.77678|26.215395|ops/ms| > |FS| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|3001.957606|29.288019|ops/ms| > |FS_ASYNC| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|123.698984|3.339233|ops/ms| > |ROCKS| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|126.252137|1.137735|ops/ms| > |ROCKS_INC| > |SerializationFrameworkMiniBenchmarks.serializerAvro|thrpt|1|30|323.658098|5.855697|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerKryo|thrpt|1|30|183.34423|3.710787|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerPojo|thrpt|1|30|404.380233|5.131744|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerRow|thrpt|1|30|527.193369|10.176726|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerTuple|thrpt|1|30|550.073024|11.724412|ops/ms| > | | > |StreamNetworkBroadcastThroughputBenchmarkExecutor.networkBroadcastThroughput|thrpt|1|30|564.690627|13.766809|ops/ms| > | | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|49918.11806|2324.234776|ops/ms|100,100ms| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|10443.63491|315.835962|ops/ms|100,100ms,SSL| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|21387.47608|2779.832704|ops/ms|1000,1ms| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|26585.85453|860.243347|ops/ms|1000,100ms| > | > |StreamNetworkThroughputBenchmarkExecutor.networkThroughput|thrpt|1|30|8252.563405|947.129028|ops/ms|1000,100ms,SSL| > | > |SumLongsBenchmark.benchmarkCount|thrpt|1|30|8806.021402|263.995836|ops/ms| | > | > |WindowBenchmarks.globalWindow|thrpt|1|30|4573.620126|112.099391|ops/ms| | | > |WindowBenchmarks.sessionWindow|thrpt|1|30|585.246412|7.026569|ops/ms| | | > |WindowBenchmarks.slidingWindow|thrpt|1|30|449.302134|4.123669|ops/ms| | | > |WindowBenchmarks.tumblingWindow|thrpt|1|30|2979.806858|33.818909|ops/ms| | | > |StreamNetworkLatencyBenchmarkExecutor.networkLatency1to1|avgt|1|30|12.842865|0.13796|ms/op| > | | > Result without the optimization: > > |Benchmark|Mode|Threads|Samples|Score|Score Error (99.9%)|Unit|Param: > channelsFlushTimeout|Param: stateBackend| > |KeyByBenchmarks.arrayKeyBy|thrpt|1|30|2060.241715|59.898485|ops/ms| | | > |KeyByBenchmarks.tupleKeyBy|thrpt|1|30|3645.306819|223.821719|ops/ms| | | > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2992.698822|36.978115|ops/ms| > |MEMORY| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2756.10949|27.798937|ops/ms| > |FS| > |MemoryStateBackendBenchmark.stateBackends|thrpt|1|30|2965.969876|44.159793|ops/ms| > |FS_ASYNC| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|125.506942|1.245978|ops/ms| > |ROCKS| > |RocksStateBackendBenchmark.stateBackends|thrpt|1|30|127.258737|1.190588|ops/ms| > |ROCKS_INC| > |SerializationFrameworkMiniBenchmarks.serializerAvro|thrpt|1|30|316.497954|8.309241|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerKryo|thrpt|1|30|189.065149|6.302073|ops/ms| > | | > |SerializationFrameworkMiniBenchmarks.serializerPojo|thrpt|1|30|391.51305|7.750728|ops/ms| > | | > |SerializationFrameworkMiniBen
[GitHub] [flink] flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-526715445 ## CI report: * dc9cdc1364b9050c3d2f3cba062310ff68d83b8e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125277359) * 5893967f343a3d2e3b4c92122303e00b01b0f8ff : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125282063) * ca8f65756da270da937077a2553a9506066b36bc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125303705) * fc346331b4e7460c75c2e4eb0eefd5f1d1bd7dc9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125561291) * 3156f2214291b0b18807c50675f7ac6b15e42ba4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125817723) * 66ba043a392e7be15c7aed2f7e2edcaef1472c15 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125818539) * df82b55379806ac8502ef92999a1abc7f9a0056b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/126191145) * 9fb1665010c88288eb49c5a9c6d1d2b81e257d71 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128271616) * 9abb55b018585e11703afe9f2494ae95fb129c0a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128277184) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wsry opened a new pull request #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly.
wsry opened a new pull request #9710: [FLINK-11859][runtime]Improve SpanningRecordSerializer performance by serializing record length to data buffer directly. URL: https://github.com/apache/flink/pull/9710 ## What is the purpose of the change The purpose of this pr is to improve the performance of SpanningRecordSerializer. Currently, the data and length filed of a serialized record are stored separately in two buffer (the lengthBuffer and the serializationBuffer), thus need two times of copy when transferring the intermediate data to BufferBuilder. This pr tries to optimize the serialization process by removing the lengthBuffer and write the length field to serializationBuffer directly, which can avoid the copy of length buffer. ## Brief change log - *Remove the length buffer of SpanningRecordSerializer and serialize the record length to data buffer directly. More specifically, the initial 4 bytes of the data buffer is reserved for length field and after the the serialization of record, the reserved space will be filled with record length.* ## Verifying this change - This change is already covered by existing tests, such as *SpanningRecordSerializerTest* and *SpanningRecordSerializationTest*. - The performance gain is proved by the micro-benchmark and the whole results can be found in this jira [FLINK-11859](https://issues.apache.org/jira/browse/FLINK-11859). ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (**yes** / no / don't know) - The runtime per-record code paths (performance sensitive): (**yes** / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] gaoyunhaii commented on a change in pull request #9679: [hotfix] Fix the comments related to side output.
gaoyunhaii commented on a change in pull request #9679: [hotfix] Fix the
comments related to side output.
URL: https://github.com/apache/flink/pull/9679#discussion_r325982849
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java
##
@@ -549,7 +549,7 @@ public void collect(StreamRecord record) {
@Override
public void collect(OutputTag outputTag, StreamRecord
record) {
if (this.outputTag == null ||
!this.outputTag.equals(outputTag)) {
- // we are only responsible for emitting to the
side-output specified by our
+ // we are not responsible for emitting to the
side-output specified by this
Review comment:
I think if the output tag is null, this output should be responsible for
emitting to the main output instead of one side output.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AT-Fieldless opened a new pull request #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese
AT-Fieldless opened a new pull request #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese URL: https://github.com/apache/flink/pull/9350 ## What is the purpose of the change Translate "Table API Example Walkthrough" page into Chinese. ## Brief change log - translate "Table API Example Walkthrough" page into Chinese. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KarmaGYZ commented on a change in pull request #9679: [hotfix] Fix the comments related to side output.
KarmaGYZ commented on a change in pull request #9679: [hotfix] Fix the comments
related to side output.
URL: https://github.com/apache/flink/pull/9679#discussion_r325982247
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java
##
@@ -549,7 +549,7 @@ public void collect(StreamRecord record) {
@Override
public void collect(OutputTag outputTag, StreamRecord
record) {
if (this.outputTag == null ||
!this.outputTag.equals(outputTag)) {
- // we are only responsible for emitting to the
side-output specified by our
+ // we are not responsible for emitting to the
side-output specified by this
Review comment:
I don't think the origin comment is wrong. Same to `RecordWriterOutput.java`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuyang1706 commented on a change in pull request #9184: [FLINK-13339][ml] Add an implementation of pipeline's api
xuyang1706 commented on a change in pull request #9184: [FLINK-13339][ml] Add
an implementation of pipeline's api
URL: https://github.com/apache/flink/pull/9184#discussion_r325981768
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/batchoperator/source/TableSourceBatchOp.java
##
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.batchoperator.source;
+
+import org.apache.flink.ml.batchoperator.BatchOperator;
+import org.apache.flink.table.api.Table;
+
+/**
+ * Transform the Table to SourceBatchOp.
+ */
+public final class TableSourceBatchOp extends BatchOperator
{
+
+ public TableSourceBatchOp(Table table) {
+ super(null);
+ if (null == table) {
+ throw new RuntimeException();
+ }
+ this.setOutput(table);
+ }
+
+ @Override
+ public TableSourceBatchOp linkFrom(BatchOperator... inputs) {
+ throw new UnsupportedOperationException("Not supported.");
Review comment:
Thanks, this message is more concise.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
carp84 commented on a change in pull request #9501: [FLINK-12697] [State
Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#discussion_r325980641
##
File path:
flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/CopyOnWriteSkipListStateMap.java
##
@@ -0,0 +1,1527 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.heap;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.ByteBufferUtils;
+import org.apache.flink.runtime.state.StateEntry;
+import org.apache.flink.runtime.state.StateTransformationFunction;
+import org.apache.flink.runtime.state.heap.space.Allocator;
+import org.apache.flink.runtime.state.heap.space.Chunk;
+import org.apache.flink.runtime.state.heap.space.SpaceUtils;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.ResourceGuard;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.Spliterators;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.flink.runtime.state.heap.SkipListUtils.HEAD_NODE;
+import static org.apache.flink.runtime.state.heap.SkipListUtils.NIL_NODE;
+import static
org.apache.flink.runtime.state.heap.SkipListUtils.NIL_VALUE_POINTER;
+
+/**
+ * Implementation of state map which is based on skip list with copy-on-write
support. states will
+ * be serialized to bytes and stored in the space allocated with the given
allocator.
+ *
+ * @param type of key
+ * @param type of namespace
+ * @param type of state
+ */
+public class CopyOnWriteSkipListStateMap extends StateMap
implements AutoCloseable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CopyOnWriteSkipListStateMap.class);
+
+ /**
+* Default max number of logically-removed keys to delete one time.
+*/
+ private static final int DEFAULT_MAX_KEYS_TO_DELETE_ONE_TIME = 3;
+
+ /**
+* Default ratio of the logically-removed keys to trigger deletion when
snapshot.
+*/
+ private static final float DEFAULT_LOGICAL_REMOVED_KEYS_RATIO = 0.2f;
+
+ /**
+* The serializer used to serialize the key and namespace to bytes
stored in skip list.
+*/
+ private final SkipListKeySerializer skipListKeySerializer;
+
+ /**
+* The serializer used to serialize the state to bytes stored in skip
list.
+*/
+ private final SkipListValueSerializer skipListValueSerializer;
+
+ /**
+* Space allocator.
+*/
+ private final Allocator spaceAllocator;
+
+ /**
+* The level index header.
+*/
+ private final LevelIndexHeader levelIndexHeader;
+
+ /**
+* Seed to generate random index level.
+*/
+ private int randomSeed;
+
+ /**
+* The current version of this map. Used for copy-on-write mechanics.
+*/
+ private int stateMapVersion;
+
+ /**
+* The highest version of this map that is still required by any
unreleased snapshot.
+*/
+ private int highestRequiredSnapshotVersion;
+
+ /**
+* Snapshots no more than this version must have been finished, but
there may be some
+* snapshots more than this version are still running.
+*/
+ private volatile int highestFinishedSnapshotVersion;
+
+ /**
+* Maintains an ordered set of version ids that are still us
[GitHub] [flink] flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-526715445 ## CI report: * dc9cdc1364b9050c3d2f3cba062310ff68d83b8e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125277359) * 5893967f343a3d2e3b4c92122303e00b01b0f8ff : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125282063) * ca8f65756da270da937077a2553a9506066b36bc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125303705) * fc346331b4e7460c75c2e4eb0eefd5f1d1bd7dc9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125561291) * 3156f2214291b0b18807c50675f7ac6b15e42ba4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125817723) * 66ba043a392e7be15c7aed2f7e2edcaef1472c15 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125818539) * df82b55379806ac8502ef92999a1abc7f9a0056b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/126191145) * 9fb1665010c88288eb49c5a9c6d1d2b81e257d71 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128271616) * 9abb55b018585e11703afe9f2494ae95fb129c0a : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] liupc commented on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/…
liupc commented on issue #9683: [FLINK-14037]Use same userCodeClassLoader as jobManager in jobClient to fix input/… URL: https://github.com/apache/flink/pull/9683#issuecomment-532955334 @TisonKun @aljoscha Thanks for reply, I will update as commented. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zzsmdfj commented on issue #9705: [hotfix][comment]remove useless comment
zzsmdfj commented on issue #9705: [hotfix][comment]remove useless comment URL: https://github.com/apache/flink/pull/9705#issuecomment-532953571 > Thanks for the hotfix, but IMO removed comment is not useless. > -1 for this pr. that comment is just meaning for "session windows UTTests" ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KurtYoung commented on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate
KurtYoung commented on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate URL: https://github.com/apache/flink/pull/8468#issuecomment-532953494 Never mind, it's me didn't explain my thought clear enough. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KarmaGYZ commented on issue #9705: [hotfix][comment]remove useless comment
KarmaGYZ commented on issue #9705: [hotfix][comment]remove useless comment URL: https://github.com/apache/flink/pull/9705#issuecomment-532950384 Thanks for the hotfix, but IMO removed comment is not useless. -1 for this pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14010) Dispatcher & JobManagers don't give up leadership when AM is shut down
[
https://issues.apache.org/jira/browse/FLINK-14010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933030#comment-16933030
]
TisonKun commented on FLINK-14010:
--
[~trohrmann] Technically I agree that it is a valid solution. Give it another
look I think we can complete shutdown future exceptionally "ResourceManager got
closed when DispatcherResourceManagerComponent is running". It infers that the
application goes into an UNKNOWN state so that the semantic is also correct.
> Dispatcher & JobManagers don't give up leadership when AM is shut down
> --
>
> Key: FLINK-14010
> URL: https://issues.apache.org/jira/browse/FLINK-14010
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Coordination
>Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
>Reporter: TisonKun
>Priority: Critical
>
> In YARN deployment scenario, YARN RM possibly launches a new AM for the job
> even if the previous AM does not terminated, for example, when AMRM heartbeat
> timeout. This is a common case that RM will send a shutdown request to the
> previous AM and expect the AM shutdown properly.
> However, currently in {{YARNResourceManager}}, we handle this request in
> {{onShutdownRequest}} which simply close the {{YARNResourceManager}} *but not
> Dispatcher and JobManagers*. Thus, Dispatcher and JobManager launched in new
> AM cannot be granted leadership properly. Visually,
> on previous AM: Dispatcher leader, JM leaders
> on new AM: ResourceManager leader
> since on client side or in per-job mode, JobManager address and port are
> configured as the new AM, the whole cluster goes into an unrecoverable
> inconsistent status: client all queries the dispatcher on new AM who is now
> the leader. Briefly, Dispatcher and JobManagers on previous AM do not give up
> their leadership properly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush
[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933023#comment-16933023 ] Jark Wu commented on FLINK-14118: - Is it only affect to 1.9? Should we also fix this in 1.8 and 1.7 series? cc [~trohrmann] > Reduce the unnecessary flushing when there is no data available for flush > - > > Key: FLINK-14118 > URL: https://issues.apache.org/jira/browse/FLINK-14118 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Yingjie Cao >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 10m > Remaining Estimate: 0h > > The new flush implementation which works by triggering a netty user event may > cause performance regression compared to the old synchronization-based one. > More specifically, when there is exactly one BufferConsumer in the buffer > queue of subpartition and no new data will be added for a while in the future > (may because of just no input or the logic of the operator is to collect some > data for processing and will not emit records immediately), that is, there is > no data to send, the OutputFlusher will continuously notify data available > and wake up the netty thread, though no data will be returned by the > pollBuffer method. > For some of our production jobs, this will incur 20% to 40% CPU overhead > compared to the old implementation. We tried to fix the problem by checking > if there is new data available when flushing, if there is no new data, the > netty thread will not be notified. It works for our jobs and the cpu usage > falls to previous level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14112) Removing zookeeper state should cause the task manager and job managers to restart
[
https://issues.apache.org/jira/browse/FLINK-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933022#comment-16933022
]
TisonKun commented on FLINK-14112:
--
I agree with [~trohrmann]'s comments.
Another question I notice is that for what reason we notify a "null"
address/session-id? I think the timeout logic can be handled by heartbeats and
if we enforce the notification always contains valid leader info we can reduce
noisy & meaningless log also simplify logic in {{LeaderRetrievalListener}}
> Removing zookeeper state should cause the task manager and job managers to
> restart
> --
>
> Key: FLINK-14112
> URL: https://issues.apache.org/jira/browse/FLINK-14112
> Project: Flink
> Issue Type: Wish
> Components: Runtime / Coordination
>Affects Versions: 1.8.1, 1.9.0
>Reporter: Aaron Levin
>Priority: Minor
>
> Suppose you have a flink application running on a cluster with the following
> configuration:
> {noformat}
> high-availability.zookeeper.path.root: /flink
> {noformat}
> Now suppose you delete all the znodes within {{/flink}}. I experienced the
> following:
> * massive amount of logging
> * application did not restart
> * task manager did not crash or restart
> * job manager did not crash or restart
> From this state I had to restart all the task managers and all the job
> managers in order for the flink application to recover.
> It would be desirable for the Task Managers and Job Managers to crash if the
> znode is not available (though perhaps you all have thought about this more
> deeply than I!)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] HuangZhenQiu commented on issue #9689: [FLINK-7151] add a basic function ddl
HuangZhenQiu commented on issue #9689: [FLINK-7151] add a basic function ddl URL: https://github.com/apache/flink/pull/9689#issuecomment-532946316 @suez1224 @fhueske @danny0405 Would you please take a look the PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AT-Fieldless commented on issue #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese
AT-Fieldless commented on issue #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese URL: https://github.com/apache/flink/pull/9350#issuecomment-532942135 > Hi @AT-Fieldless , do you have time to update this PR? Yes, I will create a new branch and update this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AT-Fieldless closed pull request #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese
AT-Fieldless closed pull request #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese URL: https://github.com/apache/flink/pull/9350 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AT-Fieldless commented on issue #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese
AT-Fieldless commented on issue #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese URL: https://github.com/apache/flink/pull/9350#issuecomment-532941960 > Hi @AT-Fieldless , do you have time to update this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AT-Fieldless removed a comment on issue #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese
AT-Fieldless removed a comment on issue #9350: [FLINK-13485] [chinese-translation] Translate "Table API Example Walkthrough" page into Chinese URL: https://github.com/apache/flink/pull/9350#issuecomment-532941960 > Hi @AT-Fieldless , do you have time to update this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-14099) SQL supports timestamp in Long
[
https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933009#comment-16933009
]
Zijie Lu edited comment on FLINK-14099 at 9/19/19 2:44 AM:
---
I have tried what you mentioned, but it seems that the timestamp field also
needs to be standard SQL timestamp and the following exception is throwed.
[~jark]
{quote}Exception in thread "main" org.apache.flink.table.api.TableException:
findAndCreateTableSource failed.Exception in thread "main"
org.apache.flink.table.api.TableException: findAndCreateTableSource failed. at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:67)
at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:54)
at
org.apache.flink.table.descriptors.ConnectTableDescriptor.registerTableSource(ConnectTableDescriptor.java:69)
at sql.job.ms.ClickStats.main(ClickStats.java:73)Caused by:
org.apache.flink.table.api.ValidationException: Rowtime attribute 'timestamp'
is not of type SQL_TIMESTAMP. at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceBase.validateRowtimeAttributeDescriptors(KafkaTableSourceBase.java:313)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceBase.(KafkaTableSourceBase.java:118)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSource.(KafkaTableSource.java:66)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactory.createKafkaTableSource(KafkaTableSourceSinkFactory.java:66)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.createStreamTableSource(KafkaTableSourceSinkFactoryBase.java:147)
at
org.apache.flink.table.factories.StreamTableSourceFactory.createTableSource(StreamTableSourceFactory.java:49)
at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:65)
... 3 more
{quote}
was (Author: alfredlu):
I have tried what you mentioned, but it seems that the timestamp field also
needs to be standard SQL timestamp and the following exception is throwed.
{quote}Exception in thread "main" org.apache.flink.table.api.TableException:
findAndCreateTableSource failed.Exception in thread "main"
org.apache.flink.table.api.TableException: findAndCreateTableSource failed. at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:67)
at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:54)
at
org.apache.flink.table.descriptors.ConnectTableDescriptor.registerTableSource(ConnectTableDescriptor.java:69)
at sql.job.ms.ClickStats.main(ClickStats.java:73)Caused by:
org.apache.flink.table.api.ValidationException: Rowtime attribute 'timestamp'
is not of type SQL_TIMESTAMP. at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceBase.validateRowtimeAttributeDescriptors(KafkaTableSourceBase.java:313)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceBase.(KafkaTableSourceBase.java:118)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSource.(KafkaTableSource.java:66)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactory.createKafkaTableSource(KafkaTableSourceSinkFactory.java:66)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.createStreamTableSource(KafkaTableSourceSinkFactoryBase.java:147)
at
org.apache.flink.table.factories.StreamTableSourceFactory.createTableSource(StreamTableSourceFactory.java:49)
at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:65)
... 3 more
{quote}
> SQL supports timestamp in Long
> --
>
> Key: FLINK-14099
> URL: https://issues.apache.org/jira/browse/FLINK-14099
> Project: Flink
> Issue Type: New Feature
>Reporter: Zijie Lu
>Priority: Major
>
> The rowtime only supports sql timestamp but not long. Can the rowtime field
> supports long?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14099) SQL supports timestamp in Long
[
https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933009#comment-16933009
]
Zijie Lu commented on FLINK-14099:
--
I have tried what you mentioned, but it seems that the timestamp field also
needs to be standard SQL timestamp and the following exception is throwed.
{quote}Exception in thread "main" org.apache.flink.table.api.TableException:
findAndCreateTableSource failed.Exception in thread "main"
org.apache.flink.table.api.TableException: findAndCreateTableSource failed. at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:67)
at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:54)
at
org.apache.flink.table.descriptors.ConnectTableDescriptor.registerTableSource(ConnectTableDescriptor.java:69)
at sql.job.ms.ClickStats.main(ClickStats.java:73)Caused by:
org.apache.flink.table.api.ValidationException: Rowtime attribute 'timestamp'
is not of type SQL_TIMESTAMP. at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceBase.validateRowtimeAttributeDescriptors(KafkaTableSourceBase.java:313)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceBase.(KafkaTableSourceBase.java:118)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSource.(KafkaTableSource.java:66)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactory.createKafkaTableSource(KafkaTableSourceSinkFactory.java:66)
at
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.createStreamTableSource(KafkaTableSourceSinkFactoryBase.java:147)
at
org.apache.flink.table.factories.StreamTableSourceFactory.createTableSource(StreamTableSourceFactory.java:49)
at
org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:65)
... 3 more
{quote}
> SQL supports timestamp in Long
> --
>
> Key: FLINK-14099
> URL: https://issues.apache.org/jira/browse/FLINK-14099
> Project: Flink
> Issue Type: New Feature
>Reporter: Zijie Lu
>Priority: Major
>
> The rowtime only supports sql timestamp but not long. Can the rowtime field
> supports long?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-526715445 ## CI report: * dc9cdc1364b9050c3d2f3cba062310ff68d83b8e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125277359) * 5893967f343a3d2e3b4c92122303e00b01b0f8ff : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125282063) * ca8f65756da270da937077a2553a9506066b36bc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125303705) * fc346331b4e7460c75c2e4eb0eefd5f1d1bd7dc9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125561291) * 3156f2214291b0b18807c50675f7ac6b15e42ba4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125817723) * 66ba043a392e7be15c7aed2f7e2edcaef1472c15 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125818539) * df82b55379806ac8502ef92999a1abc7f9a0056b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/126191145) * 9fb1665010c88288eb49c5a9c6d1d2b81e257d71 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128271616) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14093) Java8 lambdas and exceptions lead to compile error
[
https://issues.apache.org/jira/browse/FLINK-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932996#comment-16932996
]
zzsmdfj commented on FLINK-14093:
-
[~Tison] yes, you are right. I will start a discussion on dev mailing list.
> Java8 lambdas and exceptions lead to compile error
> --
>
> Key: FLINK-14093
> URL: https://issues.apache.org/jira/browse/FLINK-14093
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.0
>Reporter: zzsmdfj
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> when compile flink master by using Java 1.8.0_77, got errors as follow:
> {code:java}
> // code placeholder
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:
> compile (default-compile) on project flink-table-api-java: Compilation failure
> /home/*/zzsmdfj/sflink/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/factories/Cal
> culatedTableFactory.java:[90,53] unreported exception X; must be caught or
> declared to be thrownat
> org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:213)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:146)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:117)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:81)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
> (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main
> (Launcher.java:356)
> Caused by: org.apache.maven.plugin.compiler.CompilationFailureException:
> Compilation failure
> {code}
> if using Java 1.8.0_102 to compile, it build success. it maybe a case of bug
> [JDK-8054569|https://bugs.openjdk.java.net/browse/JDK-8054569], although we
> can fix this by upgrading the JDK,but i think it is better of compatible with
> different versions of the JDK8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14033) Distributed caches are not registered in Yarn Per Job Cluster Mode
[ https://issues.apache.org/jira/browse/FLINK-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932995#comment-16932995 ] Yang Wang commented on FLINK-14033: --- [~ZhenqiuHuang] Yeah, even many users are using -yt to distribute files so that they could be accessed by all tasks. We should still support to register by `StreamExecutionEnvironment#registerCachedFile`. Just go ahead to fix this. I could help to review. > Distributed caches are not registered in Yarn Per Job Cluster Mode > -- > > Key: FLINK-14033 > URL: https://issues.apache.org/jira/browse/FLINK-14033 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.0 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Minor > > CacheFiles in StreamExecutionEnvironment is not used in Job Submission in the > Yarn per job cluster mode. Compare to the job submission in session cluster > mode that will upload distributed cache files onto http server in application > master, we should get the cache files in job graph and register them into > blob store in YarnJobClusterEntrypoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
flinkbot edited a comment on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-526715445 ## CI report: * dc9cdc1364b9050c3d2f3cba062310ff68d83b8e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125277359) * 5893967f343a3d2e3b4c92122303e00b01b0f8ff : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125282063) * ca8f65756da270da937077a2553a9506066b36bc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125303705) * fc346331b4e7460c75c2e4eb0eefd5f1d1bd7dc9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125561291) * 3156f2214291b0b18807c50675f7ac6b15e42ba4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125817723) * 66ba043a392e7be15c7aed2f7e2edcaef1472c15 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125818539) * df82b55379806ac8502ef92999a1abc7f9a0056b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/126191145) * 9fb1665010c88288eb49c5a9c6d1d2b81e257d71 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guowei Ma updated FLINK-13993: -- Remaining Estimate: 168h Original Estimate: 168h > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Fix For: 1.10.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of…
docete commented on a change in pull request #8706: [FLINK-12814][sql-client]
Support a traditional and scrolling view of…
URL: https://github.com/apache/flink/pull/8706#discussion_r325963191
##
File path:
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/result/BasicResult.java
##
@@ -50,4 +50,12 @@ public String getWebInterfaceUrl() {
}
return webInterfaceUrl;
}
+
+ /**
+* Result types.
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of…
docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of… URL: https://github.com/apache/flink/pull/8706#discussion_r325963126 ## File path: flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/config/entries/ExecutionEntry.java ## @@ -108,6 +108,10 @@ public static final String EXECUTION_CURRENT_DATABASE = "current-database"; + private static final String EXECUTION_INTERACTIVE_VIEW_ENABLED = "interactive-view.enabled"; + + private static final String EXECUTION_MAX_QUERY_TIMEOUT_MS = "max-query-timeout-ms"; Review comment: It offers an overall protection for the user or downstream projects submitting a long-running query(e.g., select * from kafakTable). IMO for DQL cases in a streaming scenario, user always expects the result is finalizable(He/She can not stare at the screen all of the day). For user, he can use CTRL+C signal. For downstream project, a timeout should be better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14119) Clean idle state for RetractableTopNFunction
[ https://issues.apache.org/jira/browse/FLINK-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-14119: Affects Version/s: 1.9.0 > Clean idle state for RetractableTopNFunction > > > Key: FLINK-14119 > URL: https://issues.apache.org/jira/browse/FLINK-14119 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.9.0 >Reporter: Jark Wu >Assignee: ALEC CHEN >Priority: Major > > We cleaned the idle state for AppendOnlyTopNFunction and > UpdatableTopNFunction, but missed this thing for RetractableTopNFunction. We > should add it to avoid unlimited state size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14119) Clean idle state for RetractableTopNFunction
[
https://issues.apache.org/jira/browse/FLINK-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932988#comment-16932988
]
Jark Wu commented on FLINK-14119:
-
Yes. We need to cleanup the idle state in {{onTimer()}} callback, just like
UpdatableTopNFunction did. I assigned this to you.
> Clean idle state for RetractableTopNFunction
>
>
> Key: FLINK-14119
> URL: https://issues.apache.org/jira/browse/FLINK-14119
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: ALEC CHEN
>Priority: Major
>
> We cleaned the idle state for AppendOnlyTopNFunction and
> UpdatableTopNFunction, but missed this thing for RetractableTopNFunction. We
> should add it to avoid unlimited state size.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-14119) Clean idle state for RetractableTopNFunction
[ https://issues.apache.org/jira/browse/FLINK-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-14119: --- Assignee: ALEC CHEN > Clean idle state for RetractableTopNFunction > > > Key: FLINK-14119 > URL: https://issues.apache.org/jira/browse/FLINK-14119 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Jark Wu >Assignee: ALEC CHEN >Priority: Major > > We cleaned the idle state for AppendOnlyTopNFunction and > UpdatableTopNFunction, but missed this thing for RetractableTopNFunction. We > should add it to avoid unlimited state size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of…
docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of… URL: https://github.com/apache/flink/pull/8706#discussion_r325960076 ## File path: flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/config/entries/ExecutionEntry.java ## @@ -108,6 +108,10 @@ public static final String EXECUTION_CURRENT_DATABASE = "current-database"; + private static final String EXECUTION_INTERACTIVE_VIEW_ENABLED = "interactive-view.enabled"; Review comment: result-mode describes how we organize the result (change-log or table). and this switch describes how we show the result to user(interactive or non-interactive). IMO they are two different dimensions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14038) ExecutionGraph deploy failed due to akka timeout
[
https://issues.apache.org/jira/browse/FLINK-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932982#comment-16932982
]
liupengcheng commented on FLINK-14038:
--
[~zhuzh] [~trohrmann] This PR not only provide gc logs, but also heapdump, I
don't think the simple docs in [Getting
Help|https://flink.apache.org/gettinghelp.html#got-an-error-message] take
place of it. In online environment, the time cost is expensive, expecially for
routine tasks. If we just provide documents, then user must rerun the
application, and also the exceptions may varies, that depends on when OOM or
long full gc happen. What's more, I think the options is not simple for user to
get these infos (gc logs, heapdump), but they are basically needed for
debugging an application.
If there are no drawbacks or regressions, I think we should made it done
already as an default options for users.
> ExecutionGraph deploy failed due to akka timeout
>
>
> Key: FLINK-14038
> URL: https://issues.apache.org/jira/browse/FLINK-14038
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0
> Environment: Flink on yarn
> Flink 1.9.0
>Reporter: liupengcheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When launching the flink application, the following error was reported, I
> downloaded the operator logs, but still have no clue. The operator logs
> provided no useful information and was cancelled directly.
> JobManager logs:
> {code:java}
> java.lang.IllegalStateException: Update task on TaskManager
> container_e860_1567429198842_571077_01_06 @ zjy-hadoop-prc-st320.bj
> (dataPort=50990) failed due to:
> at
> org.apache.flink.runtime.executiongraph.Execution.lambda$sendUpdatePartitionInfoRpcCall$14(Execution.java:1395)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.util.concurrent.CompletionException:
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka.tcp://fl...@zjy-hadoop-prc-st320.bj:62051/user/taskmanager_0#-171547157]]
> after [1 ms]. Message of type
> [org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason
> for `AskTimeoutException` is that the recipient actor didn't send a reply.
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
[GitHub] [flink] docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of…
docete commented on a change in pull request #8706: [FLINK-12814][sql-client]
Support a traditional and scrolling view of…
URL: https://github.com/apache/flink/pull/8706#discussion_r325958937
##
File path:
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
##
@@ -613,6 +624,20 @@ private void printInfo(String message) {
terminal.flush();
}
+ /**
+* Output Results in non-interactive way.
+*
+* @return
+*/
+ void printRows(final ResultDescriptor desc, long start, long end) {
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of…
docete commented on a change in pull request #8706: [FLINK-12814][sql-client]
Support a traditional and scrolling view of…
URL: https://github.com/apache/flink/pull/8706#discussion_r325959037
##
File path:
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
##
@@ -613,6 +624,20 @@ private void printInfo(String message) {
terminal.flush();
}
+ /**
+* Output Results in non-interactive way.
+*
+* @return
+*/
+ void printRows(final ResultDescriptor desc, long start, long end) {
+ final String msg = "Time taken: %.03f seconds, Fetched %d
row(s).";
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] docete commented on a change in pull request #8706: [FLINK-12814][sql-client] Support a traditional and scrolling view of…
docete commented on a change in pull request #8706: [FLINK-12814][sql-client]
Support a traditional and scrolling view of…
URL: https://github.com/apache/flink/pull/8706#discussion_r325958597
##
File path:
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
##
@@ -464,28 +465,38 @@ private void callExplain(SqlCommandCall cmdCall) {
}
private void callSelect(SqlCommandCall cmdCall) {
+ final long start = System.currentTimeMillis();
final ResultDescriptor resultDesc;
try {
resultDesc = executor.executeQuery(context,
cmdCall.operands[0]);
} catch (SqlExecutionException e) {
printExecutionException(e);
return;
}
- final CliResultView view;
- if (resultDesc.isMaterialized()) {
- view = new CliTableResultView(this, resultDesc);
- } else {
- view = new CliChangelogResultView(this, resultDesc);
- }
+ final long end = System.currentTimeMillis();
Review comment:
User can wait a timeout(setting by max-query-timeout-ms) or just type CTRL +
C to terminate the submitted query.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (FLINK-14122) Extend State Processor API to read ListCheckpointed operator state
Seth Wiesman created FLINK-14122: Summary: Extend State Processor API to read ListCheckpointed operator state Key: FLINK-14122 URL: https://issues.apache.org/jira/browse/FLINK-14122 Project: Flink Issue Type: Sub-task Components: API / DataStream Affects Versions: 1.10.0 Reporter: Seth Wiesman The state processor api cannot read operator state using the ListCheckpointed interface because it requires access the JavaSerializer which is package private. Instead of making that class public, we should offer a readListCheckpointed Method to easily read this state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TisonKun commented on issue #9684: [FLINK-13992][coordination] Refactor Optional parameter in InputGateWithMetrics#updateMetrics
TisonKun commented on issue #9684: [FLINK-13992][coordination] Refactor Optional parameter in InputGateWithMetrics#updateMetrics URL: https://github.com/apache/flink/pull/9684#issuecomment-532922137 ping @azagrebin as a reminder :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14109) Improve javadocs and tests for high-availability backend
[
https://issues.apache.org/jira/browse/FLINK-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932959#comment-16932959
]
TisonKun commented on FLINK-14109:
--
[~bremac] aha, nice to hear :-)
> Improve javadocs and tests for high-availability backend
>
>
> Key: FLINK-14109
> URL: https://issues.apache.org/jira/browse/FLINK-14109
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Brendan MacDonell
>Assignee: Brendan MacDonell
>Priority: Minor
>
> My team at Sight Machine just finished building a custom HA backend for our
> infrastructure. The process was mostly painless, but we ran into a few pain
> points on the way:
> * {{CompletedCheckpointStore#getLatestCheckpoint}} is not marked as
> {{@Nullable}}, so there was some confusion about whether the Javadocs or
> (lack of) annotation is correct. The interface would be clearer if the
> annotation was present.
> * The javadocs for {{CompletedCheckpointStore#recover}} disagree with the
> documentation for {{ZooKeeperCompletedCheckpointStore#recover}}. It's not
> immediately clear to someone working on the code that the ZK javadoc is
> outdated and the interface documentation is correct.
> * -The base {{CompletedCheckpointStore}} tests only work with
> high-availability backends that keep a list of checkpoints in memory. If the
> backend persists and retrieves data from another source the tests will fail.
> It's fairly simple to change the tests to lift this requirement though. See
> [this gist|https://gist.github.com/bremac/1b3365bc0257dfbd33bcd0b7a7627c00]
> for an example.-
> We've got patches for the points above that we'd be happy to contribute. :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tweise merged pull request #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
tweise merged pull request #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9708 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tweise commented on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
tweise commented on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9708#issuecomment-532910734 unrelated test failure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tweise merged pull request #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
tweise merged pull request #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9709 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tweise commented on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
tweise commented on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9709#issuecomment-532910374 unrelated test failure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot edited a comment on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9708#issuecomment-532891597 ## CI report: * adf5d969d37c127653d77821bfa5fe48cf212a6d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128256147) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot edited a comment on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9709#issuecomment-532891619 ## CI report: * 02a0e0ea41b1dc31f233ca4d25434b5370451d1a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128256191) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot edited a comment on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9709#issuecomment-532891619 ## CI report: * 02a0e0ea41b1dc31f233ca4d25434b5370451d1a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128256191) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot edited a comment on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9708#issuecomment-532891597 ## CI report: * adf5d969d37c127653d77821bfa5fe48cf212a6d : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128256147) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot commented on issue #9708: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9708#issuecomment-532891597 ## CI report: * adf5d969d37c127653d77821bfa5fe48cf212a6d : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot commented on issue #9709: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9709#issuecomment-532891619 ## CI report: * 02a0e0ea41b1dc31f233ca4d25434b5370451d1a : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-14107) Kinesis consumer record emitter deadlock under event time alignment
[ https://issues.apache.org/jira/browse/FLINK-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved FLINK-14107. -- Fix Version/s: 1.10.0 Resolution: Fixed > Kinesis consumer record emitter deadlock under event time alignment > --- > > Key: FLINK-14107 > URL: https://issues.apache.org/jira/browse/FLINK-14107 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.8.2, 1.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When the emitter reaches the max timestamp for the current queue, it stops > emitting and waits for the max timestamp to advance. Since it simultaneously > selects the next queue as the new "minimum" queue, it may deadlock if the > previous min queue represents the new global lower bound after the max > timestamp advanced. This occurs very infrequently and we were finally able to > reproduce. -- This message was sent by Atlassian Jira (v8.3.4#803005)
