[GitHub] [flink] Myasuka commented on pull request #15941: [FLINK-22659]Adding execution.checkpointing.interval to Flink Doc
Myasuka commented on pull request #15941: URL: https://github.com/apache/flink/pull/15941#issuecomment-844769265 @lixmgl , you could use git force push to update your previous commits so that those commit messages could be updated. I just noticed that you create the PR against `release-1.13`, And actually we would always create PR against `master` branch except that too much code difference across those branches. Committer who helps to merge your PR to `master` branch would help to pick your commit to other `release-xxx` branch if necessary. Please create another PR against `master` branch and closing this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15927: [FLINK-22639][runtime] ClassLoaderUtil cannot print classpath of Flin…
flinkbot edited a comment on pull request #15927: URL: https://github.com/apache/flink/pull/15927#issuecomment-842054909 ## CI report: * ce1bf535ffd28ea2b7fc10264a184f05e8980aed Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18160) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15944: [FLINK-22694][e2e] Use sql file in TPCH end to end tests
flinkbot edited a comment on pull request #15944: URL: https://github.com/apache/flink/pull/15944#issuecomment-843049332 ## CI report: * b64a13c6b1f1c16817a40ca21a20574f51eabcfb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18085) * 1045b5d73acfd0b8dadd5601d001274ac6ed00f4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18163) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] beyond1920 opened a new pull request #15963: [FLINK-22719][table-planner-blink] Fall back to regular join instead of thrown exception if a join does not satisfy conditions to translat
beyond1920 opened a new pull request #15963: URL: https://github.com/apache/flink/pull/15963 ## What is the purpose of the change This pr aims to remove throw exception logical in `WindowJoinUtil`#`containsWindowStartEqualityAndEndEquality`. After the update, the physical node will fall back to regular join instead of thrown exception if a join does not satisfy conditions to translate into WindowJoin. ## Brief change log - remove throw exception logical in `WindowJoinUtil` ## Verifying this change - * Exists UT in `WindowJoinTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22719) WindowJoinUtil.containsWindowStartEqualityAndEndEquality should not throw exception
[ https://issues.apache.org/jira/browse/FLINK-22719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-22719: --- Labels: pull-request-available (was: ) > WindowJoinUtil.containsWindowStartEqualityAndEndEquality should not throw > exception > --- > > Key: FLINK-22719 > URL: https://issues.apache.org/jira/browse/FLINK-22719 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > This will broke regular join sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rmetzger commented on a change in pull request #15884: [FLINK-22266] Fix stop-with-savepoint operation in AdaptiveScheduler
rmetzger commented on a change in pull request #15884:
URL: https://github.com/apache/flink/pull/15884#discussion_r635821863
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/adaptive/ExecutingTest.java
##
@@ -258,6 +305,23 @@ public void
testFalseReportsViaUpdateTaskExecutionStateAreIgnored() throws Excep
}
}
+@Test
+public void testExecutionVertexMarkedAsFailedOnDeploymentFailure() throws
Exception {
+try (MockExecutingContext ctx = new MockExecutingContext()) {
+MockExecutionJobVertex mejv =
+new
MockExecutionJobVertex(FailOnDeployMockExecutionVertex::new);
+ExecutionGraph executionGraph =
+new MockExecutionGraph(() ->
Collections.singletonList(mejv));
+Executing exec =
+new
ExecutingStateBuilder().setExecutionGraph(executionGraph).build(ctx);
+
+assertThat(
+((FailOnDeployMockExecutionVertex)
mejv.getMockExecutionVertex())
+.getMarkedFailure(),
+is(instanceOf(JobException.class)));
Review comment:
The error handling of markFailed is difficult to test, because so many
components are involved. But in my opinion, we have good test coverage:
markFailed will (through the DefaultExecutionGraph) notify the
`InternalFailuresListener` about the task failure. The
`UpdateSchedulerNgOnInternalFailuresListener` implementation used by adaptive
scheduler will call updateTaskExecutionState on the scheduler. This chain of
calls will be used for example for the failure in the
`AdaptiveSchedulerITCase.testGlobalFailoverCanRecoverState()` test.
For the Executing state, we have tests that exceptions during deployment
lead to a markFailed call
(`testExecutionVertexMarkedAsFailedOnDeploymentFailure`), and failures reported
via updateTaskExecutionState to appropriate error handling
(`testFailureReportedViaUpdateTaskExecutionStateCausesFailingOnNoRestart`,
`testFailureReportedViaUpdateTaskExecutionStateCausesRestart`,
`testFalseReportsViaUpdateTaskExecutionStateAreIgnored`).
Adding a test that a markFailed call will notify the
`InternalFailuresListener` is out of the scope of the ExecutingTest (because we
are testing the ExecutionVertex and Execution classes).
Adding a test that a markFailed call will call updateTaskExecutionState will
need to go through a test specific `InternalFailuresListener`: Since all the
relevant calls on ExecutingState are already covered, this would only test the
test specific `InternalFailuresListener`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15963: [FLINK-22719][table-planner-blink] Fall back to regular join instead of thrown exception if a join does not satisfy conditions to translate
flinkbot commented on pull request #15963: URL: https://github.com/apache/flink/pull/15963#issuecomment-844772764 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit da47dc6ac7b349a1c7fccb37da5028f02203706e (Thu May 20 07:05:58 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-22719).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22693) EXPLAIN cannot be used on Hbase table when useing ROW type
[ https://issues.apache.org/jira/browse/FLINK-22693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348114#comment-17348114 ] sadf commented on FLINK-22693: -- I guess from the error exception that the type does not match, so I made the following attempt To make the type match, I changed this class org.apache.flink.table.planner.calcite.FlinkTypeFactory to let it return StructKind.NONE ,but it didn't work Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 60 to line 1, column 70: Cannot assign to target field 'f1' of type RecordType(INTEGER name) from source field 'f1' of type RecordType(INTEGER EXPR$0)Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 60 to line 1, column 70: Cannot assign to target field 'f1' of type RecordType(INTEGER name) from source field 'f1' of type RecordType(INTEGER EXPR$0) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:467) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:883) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:868) at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5043) at org.apache.calcite.sql.validate.SqlValidatorImpl.checkTypeAssignment(SqlValidatorImpl.java:4714) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateInsert(SqlValidatorImpl.java:4417) at org.apache.calcite.sql.SqlInsert.validate(SqlInsert.java:158) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1016) > EXPLAIN cannot be used on Hbase table when useing ROW type > -- > > Key: FLINK-22693 > URL: https://issues.apache.org/jira/browse/FLINK-22693 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase, Table SQL / API >Affects Versions: 1.12.2 >Reporter: sadf >Priority: Major > Attachments: Driver.java, hbase.PNG, pom.xml > > > We use ’EXPLAIN PLAN FOR‘ as a way of validating SQL > calcite will no longer work and throw an exception when we sink to HBase > table and use row type > !hbase.PNG! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rmetzger commented on a change in pull request #15884: [FLINK-22266] Fix stop-with-savepoint operation in AdaptiveScheduler
rmetzger commented on a change in pull request #15884:
URL: https://github.com/apache/flink/pull/15884#discussion_r635823536
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/Executing.java
##
@@ -59,9 +69,17 @@
super(context, executionGraph, executionGraphHandler,
operatorCoordinatorHandler, logger);
this.context = context;
this.userCodeClassLoader = userCodeClassLoader;
+Preconditions.checkState(
+executionGraph.getState() == JobStatus.RUNNING, "Assuming
running execution graph");
-deploy();
-
+if (executingStateBehavior == Behavior.DEPLOY_ON_ENTER) {
+onAllExecutionVertexes(this::deploySafely);
+} else if (executingStateBehavior == Behavior.EXPECT_RUNNING) {
+onAllExecutionVertexes(this::expectRunning);
+} else {
+throw new IllegalStateException(
+"Unexpected executing state behavior " +
executingStateBehavior);
+}
Review comment:
Indeed, that's a good simplification. I pushed a commit addressing this
item.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] lixmgl closed pull request #15941: [FLINK-22659]Adding execution.checkpointing.interval to Flink Doc
lixmgl closed pull request #15941: URL: https://github.com/apache/flink/pull/15941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22701) "bin/flink savepoint -d :savepointpath" cannot locate savepointpath arg
[
https://issues.apache.org/jira/browse/FLINK-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348119#comment-17348119
]
Yun Tang commented on FLINK-22701:
--
Could you paste the actual command you used (do not use {{:savepointpath}} to
replace your real path)?
> "bin/flink savepoint -d :savepointpath" cannot locate savepointpath arg
> ---
>
> Key: FLINK-22701
> URL: https://issues.apache.org/jira/browse/FLINK-22701
> Project: Flink
> Issue Type: Bug
> Components: Command Line Client
>Affects Versions: 1.12.2
> Environment: Kubernetes standalone cluster
>Reporter: Jinliang Guo
>Priority: Major
>
> Hi there,
>
> When I use "bin/flink savepoint -d :savepointpath" to dispose a savepoint,
> got following exceptions:
> The program finished with the following
> exception:java.lang.NullPointerException: Missing required argument:
> savepoint path. Usage: bin/flink savepoint -d
> at
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:76)
> at
> org.apache.flink.client.cli.CliFrontend.disposeSavepoint(CliFrontend.java:785)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$savepoint$8(CliFrontend.java:723)
> at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
> at
> org.apache.flink.client.cli.CliFrontend.savepoint(CliFrontend.java:719)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1072)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> However, if I use "bin/flink savepoint -dispose :savepointpath", it works
> good.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] fapaul opened a new pull request #15964: [BP-1.13][FLINK-22434] Store suspended execution graphs on termination to keep them accessible
fapaul opened a new pull request #15964: URL: https://github.com/apache/flink/pull/15964 Unchanged backport of #15799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22437) Miss adding parallesim for filter operator in batch mode
[
https://issues.apache.org/jira/browse/FLINK-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348120#comment-17348120
]
godfrey he commented on FLINK-22437:
[~zoucao] would you like to take this ticket ?
> Miss adding parallesim for filter operator in batch mode
>
>
> Key: FLINK-22437
> URL: https://issues.apache.org/jira/browse/FLINK-22437
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.2
>Reporter: zoucao
>Priority: Major
>
> when I execute batch sql as follow in flink-1.12.2, I found lots of small
> files in hdfs. In filesystem connector, `GroupedPartitionWriter` will be
> used, and it close the last partiton if a new record does not belong to the
> existing partition. The phenomenon occurred if there are more than one
> partiton's records are sent to filesystem sink at the same time. Hive source
> can determine parallesim by the number of file and partition, and the
> parallesim will extended by sort operator, but in
> `CommonPhysicalSink#createSinkTransformation`，a filter operator will be add
> to support `SinkNotNullEnforcer`, there is no parallesim set for it, so
> filesystem sink operator can not get the correct parallesim from inputstream.
> {code:java}
> CREATE CATALOG myHive with (
> 'type'='hive',
> 'property-version'='1',
> 'default-database' = 'flink_sql_online_test'
> );
> -- SET table.sql-dialect=hive;
> -- CREATE TABLE IF NOT EXISTS myHive.flink_sql_online_test.hive_sink (
> --`timestamp` BIGINT,
> --`time` STRING,
> --id BIGINT,
> --product STRING,
> --price DOUBLE,
> --canSell STRING,
> --selledNum BIGINT
> -- ) PARTITIONED BY (
> --dt STRING,
> --`hour` STRING,
> -- `min` STRING
> -- ) TBLPROPERTIES (
> --'partition.time-extractor.timestamp-pattern'='$dt $hr:$min:00',
> --'sink.partition-commit.trigger'='partition-time',
> --'sink.partition-commit.delay'='1 min',
> --'sink.partition-commit.policy.kind'='metastore,success-file'
> -- );
> create table fs_sink (
> `timestamp` BIGINT,
> `time` STRING,
> id BIGINT,
> product STRING,
> price DOUBLE,
> canSell STRING,
> selledNum BIGINT,
> dt STRING,
> `hour` STRING,
> `min` STRING
> ) PARTITIONED BY (dt, `hour`, `min`) with (
> 'connector'='filesystem',
> 'path'='hdfs://',
> 'format'='csv'
> );
> insert into fs_sink
> select * from myHive.flink_sql_online_test.hive_sink;
> {code}
> I think this problem can be fixed by adding a parallesim for it just like
> {code:java}
> val dataStream = new DataStream(env, inputTransformation).filter(enforcer)
> .setParallelism(inputTransformation.getParallelism)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] fhan688 opened a new pull request #15965: [datagen] Normalize parameter names in RandomGeneratorVisitor and SequenceGeneratorVisitor
fhan688 opened a new pull request #15965: URL: https://github.com/apache/flink/pull/15965 ## What is the purpose of the change This PR normalized parameter names in RandomGeneratorVisitor and SequenceGeneratorVisitor. related methods:  ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-22719) WindowJoinUtil.containsWindowStartEqualityAndEndEquality should not throw exception
[ https://issues.apache.org/jira/browse/FLINK-22719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-22719: Assignee: Andy > WindowJoinUtil.containsWindowStartEqualityAndEndEquality should not throw > exception > --- > > Key: FLINK-22719 > URL: https://issues.apache.org/jira/browse/FLINK-22719 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Jingsong Lee >Assignee: Andy >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > This will broke regular join sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] lixmgl opened a new pull request #15966: [FLINK-22659][docs] Add execution.checkpointing.interval to configura…
lixmgl opened a new pull request #15966: URL: https://github.com/apache/flink/pull/15966 …tion docs ## What is the purpose of the change *Adding execution.checkpointing.interval to Flink configuration docs* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* **This change is a trivial rework / code cleanup without any test coverage.** *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15965: [datagen] Normalize parameter names in RandomGeneratorVisitor and SequenceGeneratorVisitor
flinkbot commented on pull request #15965: URL: https://github.com/apache/flink/pull/15965#issuecomment-844792525 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9ab417c63ef4d6a0f4f3b785980914395901e955 (Thu May 20 07:23:25 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15964: [BP-1.13][FLINK-22434] Store suspended execution graphs on termination to keep them accessible
flinkbot commented on pull request #15964: URL: https://github.com/apache/flink/pull/15964#issuecomment-844792577 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c2550460db2c0951911dd3219190d0f59e9d1ac6 (Thu May 20 07:23:28 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-22659) 'execution.checkpointing.interval' missing in Flink doc
[
https://issues.apache.org/jira/browse/FLINK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346650#comment-17346650
]
Rainie Li edited comment on FLINK-22659 at 5/20/21, 7:23 AM:
-
Please review https://github.com/apache/flink/pull/15966
was (Author: rainieli):
Please review [https://github.com/apache/flink/pull/15941]
> 'execution.checkpointing.interval' missing in Flink doc
>
>
> Key: FLINK-22659
> URL: https://issues.apache.org/jira/browse/FLINK-22659
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.13.0
>Reporter: wangqinghuan
>Assignee: Rainie Li
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Flink deployment configuration describe how to configure checkpointing in
> flink-conf.yaml[https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/]
> {quote}*Checkpointing*
> You can configure checkpointing directly in code within your Flink job or
> application. Putting these values here in the configuration defines them as
> defaults in case the application does not configure anything.
> {quote}
> *
>
> {quote}{{state.backend}}: The state backend to use. This defines the data
> structure mechanism for taking snapshots. Common values are {{filesystem}} or
> {{rocksdb}}.
> {quote} *
> {quote}{{state.checkpoints.dir}}: The directory to write checkpoints to. This
> takes a path URI like _s3://mybucket/flink-app/checkpoints_ or
> _hdfs://namenode:port/flink/checkpoints_.
> {quote}
> *
> {quote}{{state.savepoints.dir}}: The default directory for savepoints. Takes
> a path URI, similar to {{state.checkpoints.dir}}.
> {quote}
> In my test for Flink-1.13.0, however,Flink checkpointing was not enabled
> without 'execution.checkpointing.interval' value in flink-conf.yaml. In order
> to enable checkpointing in case the application does not configure anything,
> we need to configure these values in flink-conf.yaml.
> * {{state.backend}}:
> * {{state.checkpoints.dir:}}
> * {{state.savepoints.dir:}}
> * execution.checkpointing.interval:
> {{'execution.checkpointing.interval' value missing in document.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] lixmgl commented on pull request #15941: [FLINK-22659]Adding execution.checkpointing.interval to Flink Doc
lixmgl commented on pull request #15941: URL: https://github.com/apache/flink/pull/15941#issuecomment-844793503 thanks @Myasuka here is the new PR: https://github.com/apache/flink/pull/15966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15966: [FLINK-22659][docs] Add execution.checkpointing.interval to configura…
flinkbot commented on pull request #15966: URL: https://github.com/apache/flink/pull/15966#issuecomment-844795394 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 50986672248961509dc127d1df42ec8ef96b5ecd (Thu May 20 07:25:39 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15808: [FLINK-21229] add confluent schema registry ssl support
flinkbot edited a comment on pull request #15808: URL: https://github.com/apache/flink/pull/15808#issuecomment-829227618 ## CI report: * c55980ce590f38d3f0cd23ac97bd331d1a1c3b15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18148) * 0494b5f517497eda59a8c75748c1ece64f73a0d9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15884: [FLINK-22266] Fix stop-with-savepoint operation in AdaptiveScheduler
flinkbot edited a comment on pull request #15884: URL: https://github.com/apache/flink/pull/15884#issuecomment-836859750 ## CI report: * 94a973c0c9f5daa60afa36d7bec4ff2b811d9c7b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18033) * 8f0b36f54fa0e38dc83565eb2d5a8c4ce204fb82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15963: [FLINK-22719][table-planner-blink] Fall back to regular join instead of thrown exception if a join does not satisfy conditions to translate
flinkbot commented on pull request #15963: URL: https://github.com/apache/flink/pull/15963#issuecomment-844797710 ## CI report: * 2a5da984eedad7dcdef215e093c80e89e6658491 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache closed pull request #15920: [FLINK-22661][hive] HiveInputFormatPartitionReader can return invalid…
lirui-apache closed pull request #15920: URL: https://github.com/apache/flink/pull/15920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22437) Miss adding parallesim for filter operator in batch mode
[
https://issues.apache.org/jira/browse/FLINK-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348127#comment-17348127
]
zoucao commented on FLINK-22437:
hi [~godfreyhe], thanks for your reply, I am glad to take it. By the way, I
think it can be easily fixed by add Parallelism for filter operator, WDYS ？
> Miss adding parallesim for filter operator in batch mode
>
>
> Key: FLINK-22437
> URL: https://issues.apache.org/jira/browse/FLINK-22437
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.2
>Reporter: zoucao
>Priority: Major
>
> when I execute batch sql as follow in flink-1.12.2, I found lots of small
> files in hdfs. In filesystem connector, `GroupedPartitionWriter` will be
> used, and it close the last partiton if a new record does not belong to the
> existing partition. The phenomenon occurred if there are more than one
> partiton's records are sent to filesystem sink at the same time. Hive source
> can determine parallesim by the number of file and partition, and the
> parallesim will extended by sort operator, but in
> `CommonPhysicalSink#createSinkTransformation`，a filter operator will be add
> to support `SinkNotNullEnforcer`, there is no parallesim set for it, so
> filesystem sink operator can not get the correct parallesim from inputstream.
> {code:java}
> CREATE CATALOG myHive with (
> 'type'='hive',
> 'property-version'='1',
> 'default-database' = 'flink_sql_online_test'
> );
> -- SET table.sql-dialect=hive;
> -- CREATE TABLE IF NOT EXISTS myHive.flink_sql_online_test.hive_sink (
> --`timestamp` BIGINT,
> --`time` STRING,
> --id BIGINT,
> --product STRING,
> --price DOUBLE,
> --canSell STRING,
> --selledNum BIGINT
> -- ) PARTITIONED BY (
> --dt STRING,
> --`hour` STRING,
> -- `min` STRING
> -- ) TBLPROPERTIES (
> --'partition.time-extractor.timestamp-pattern'='$dt $hr:$min:00',
> --'sink.partition-commit.trigger'='partition-time',
> --'sink.partition-commit.delay'='1 min',
> --'sink.partition-commit.policy.kind'='metastore,success-file'
> -- );
> create table fs_sink (
> `timestamp` BIGINT,
> `time` STRING,
> id BIGINT,
> product STRING,
> price DOUBLE,
> canSell STRING,
> selledNum BIGINT,
> dt STRING,
> `hour` STRING,
> `min` STRING
> ) PARTITIONED BY (dt, `hour`, `min`) with (
> 'connector'='filesystem',
> 'path'='hdfs://',
> 'format'='csv'
> );
> insert into fs_sink
> select * from myHive.flink_sql_online_test.hive_sink;
> {code}
> I think this problem can be fixed by adding a parallesim for it just like
> {code:java}
> val dataStream = new DataStream(env, inputTransformation).filter(enforcer)
> .setParallelism(inputTransformation.getParallelism)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15004: [FLINK-21253][table-planner-blink] Support grouping set syntax for GroupWindowAggregate
flinkbot edited a comment on pull request #15004: URL: https://github.com/apache/flink/pull/15004#issuecomment-785052726 ## CI report: * b826a818fcd0d47f2454575c59d324519e2c6ad9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18140) * 4b01a38ffe4968511607b84c58309bf75183085a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15808: [FLINK-21229] add confluent schema registry ssl support
flinkbot edited a comment on pull request #15808: URL: https://github.com/apache/flink/pull/15808#issuecomment-829227618 ## CI report: * c55980ce590f38d3f0cd23ac97bd331d1a1c3b15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18148) * 0494b5f517497eda59a8c75748c1ece64f73a0d9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18164) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15884: [FLINK-22266] Fix stop-with-savepoint operation in AdaptiveScheduler
flinkbot edited a comment on pull request #15884: URL: https://github.com/apache/flink/pull/15884#issuecomment-836859750 ## CI report: * 94a973c0c9f5daa60afa36d7bec4ff2b811d9c7b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18033) * 8f0b36f54fa0e38dc83565eb2d5a8c4ce204fb82 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18165) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15963: [FLINK-22719][table-planner-blink] Fall back to regular join instead of thrown exception if a join does not satisfy conditions to tra
flinkbot edited a comment on pull request #15963: URL: https://github.com/apache/flink/pull/15963#issuecomment-844797710 ## CI report: * 2a5da984eedad7dcdef215e093c80e89e6658491 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18166) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15964: [BP-1.13][FLINK-22434] Store suspended execution graphs on termination to keep them accessible
flinkbot commented on pull request #15964: URL: https://github.com/apache/flink/pull/15964#issuecomment-844823190 ## CI report: * c2550460db2c0951911dd3219190d0f59e9d1ac6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15965: [datagen] Normalize parameter names in RandomGeneratorVisitor and SequenceGeneratorVisitor
flinkbot commented on pull request #15965: URL: https://github.com/apache/flink/pull/15965#issuecomment-844823334 ## CI report: * 9ab417c63ef4d6a0f4f3b785980914395901e955 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15966: [FLINK-22659][docs] Add execution.checkpointing.interval to configura…
flinkbot commented on pull request #15966: URL: https://github.com/apache/flink/pull/15966#issuecomment-844823503 ## CI report: * 50986672248961509dc127d1df42ec8ef96b5ecd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-11310) Convert predicates to IN or NOT_IN for Project
[ https://issues.apache.org/jira/browse/FLINK-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he closed FLINK-11310. -- Resolution: Won't Fix Currently, the or expressions will be converted to Sarg, and will be put into a HashSet when code gen. So this issue is invalid now, I will close it. > Convert predicates to IN or NOT_IN for Project > -- > > Key: FLINK-11310 > URL: https://issues.apache.org/jira/browse/FLINK-11310 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Reporter: Hequn Cheng >Priority: Major > Labels: auto-unassigned, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > In FLINK-10474, we force translate IN into a predicate to avoid translating > to a JOIN. In addition, we add a Rule to convert the predicates back to IN so > that we can generate code using a HashSet for the IN. > However, FLINK-10474 only takes Filter into consideration. It would be great > to also convert predicates in Project to IN. It not only will improve the > performance for the Project, but also will avoid the problem raised in > FLINK-11308, as all predicates will be converted into one IN expression. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] yittg commented on pull request #15768: [FLINK-22451][table] Support (*) as parameter of UDFs in Table API
yittg commented on pull request #15768: URL: https://github.com/apache/flink/pull/15768#issuecomment-844825760 Great, really appreciate for you help. @wuchong -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22720) UpsertKafkaTableITCase.testAggregate fail due to ConcurrentModificationException
[
https://issues.apache.org/jira/browse/FLINK-22720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348133#comment-17348133
]
Jark Wu commented on FLINK-22720:
-
I will take this.
> UpsertKafkaTableITCase.testAggregate fail due to
> ConcurrentModificationException
>
>
> Key: FLINK-22720
> URL: https://issues.apache.org/jira/browse/FLINK-22720
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.14.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18151&view=logs&j=c5612577-f1f7-5977-6ff6-7432788526f7&t=53f6305f-55e6-561c-8f1e-3a1dde2c77df&l=6613
> {code:java}
> 2021-05-19T21:28:02.8689083Z May 19 21:28:02 [ERROR] testAggregate[format =
> avro](org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase)
> Time elapsed: 2.067 s <<< ERROR!
> 2021-05-19T21:28:02.8708337Z May 19 21:28:02
> java.util.ConcurrentModificationException
> 2021-05-19T21:28:02.8710333Z May 19 21:28:02 at
> java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
> 2021-05-19T21:28:02.8712083Z May 19 21:28:02 at
> java.util.HashMap$ValueIterator.next(HashMap.java:1474)
> 2021-05-19T21:28:02.8712680Z May 19 21:28:02 at
> java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> 2021-05-19T21:28:02.8713142Z May 19 21:28:02 at
> java.util.ArrayList.addAll(ArrayList.java:583)
> 2021-05-19T21:28:02.8716029Z May 19 21:28:02 at
> org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions.lambda$getResults$0(TestValuesRuntimeFunctions.java:114)
> 2021-05-19T21:28:02.8717007Z May 19 21:28:02 at
> java.util.HashMap$Values.forEach(HashMap.java:981)
> 2021-05-19T21:28:02.8718041Z May 19 21:28:02 at
> org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions.getResults(TestValuesRuntimeFunctions.java:114)
> 2021-05-19T21:28:02.8719339Z May 19 21:28:02 at
> org.apache.flink.table.planner.factories.TestValuesTableFactory.getResults(TestValuesTableFactory.java:184)
> 2021-05-19T21:28:02.8720309Z May 19 21:28:02 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.waitingExpectedResults(KafkaTableTestUtils.java:82)
> 2021-05-19T21:28:02.8721311Z May 19 21:28:02 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.wordFreqToUpsertKafka(UpsertKafkaTableITCase.java:440)
> 2021-05-19T21:28:02.8730402Z May 19 21:28:02 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.testAggregate(UpsertKafkaTableITCase.java:73)
> 2021-05-19T21:28:02.8731390Z May 19 21:28:02 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-19T21:28:02.8732095Z May 19 21:28:02 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-19T21:28:02.8732935Z May 19 21:28:02 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-19T21:28:02.8733726Z May 19 21:28:02 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-19T21:28:02.8734598Z May 19 21:28:02 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-19T21:28:02.8735450Z May 19 21:28:02 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-19T21:28:02.8736313Z May 19 21:28:02 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-19T21:28:02.8737329Z May 19 21:28:02 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-19T21:28:02.8738165Z May 19 21:28:02 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-05-19T21:28:02.8738989Z May 19 21:28:02 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-05-19T21:28:02.8739741Z May 19 21:28:02 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-19T21:28:02.8740563Z May 19 21:28:02 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-19T21:28:02.8741340Z May 19 21:28:02 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-19T21:28:02.8742077Z May 19 21:28:02 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-19T21:28:02.8742802Z May 19 21:28:02 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-19T21:28:02.8743594Z May 19 21:28:02 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-19T21:28:02.8744811Z May 19 21:28:02 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassR
[jira] [Assigned] (FLINK-22720) UpsertKafkaTableITCase.testAggregate fail due to ConcurrentModificationException
[
https://issues.apache.org/jira/browse/FLINK-22720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-22720:
---
Assignee: Jark Wu
> UpsertKafkaTableITCase.testAggregate fail due to
> ConcurrentModificationException
>
>
> Key: FLINK-22720
> URL: https://issues.apache.org/jira/browse/FLINK-22720
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.14.0
>Reporter: Guowei Ma
>Assignee: Jark Wu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18151&view=logs&j=c5612577-f1f7-5977-6ff6-7432788526f7&t=53f6305f-55e6-561c-8f1e-3a1dde2c77df&l=6613
> {code:java}
> 2021-05-19T21:28:02.8689083Z May 19 21:28:02 [ERROR] testAggregate[format =
> avro](org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase)
> Time elapsed: 2.067 s <<< ERROR!
> 2021-05-19T21:28:02.8708337Z May 19 21:28:02
> java.util.ConcurrentModificationException
> 2021-05-19T21:28:02.8710333Z May 19 21:28:02 at
> java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
> 2021-05-19T21:28:02.8712083Z May 19 21:28:02 at
> java.util.HashMap$ValueIterator.next(HashMap.java:1474)
> 2021-05-19T21:28:02.8712680Z May 19 21:28:02 at
> java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> 2021-05-19T21:28:02.8713142Z May 19 21:28:02 at
> java.util.ArrayList.addAll(ArrayList.java:583)
> 2021-05-19T21:28:02.8716029Z May 19 21:28:02 at
> org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions.lambda$getResults$0(TestValuesRuntimeFunctions.java:114)
> 2021-05-19T21:28:02.8717007Z May 19 21:28:02 at
> java.util.HashMap$Values.forEach(HashMap.java:981)
> 2021-05-19T21:28:02.8718041Z May 19 21:28:02 at
> org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions.getResults(TestValuesRuntimeFunctions.java:114)
> 2021-05-19T21:28:02.8719339Z May 19 21:28:02 at
> org.apache.flink.table.planner.factories.TestValuesTableFactory.getResults(TestValuesTableFactory.java:184)
> 2021-05-19T21:28:02.8720309Z May 19 21:28:02 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.waitingExpectedResults(KafkaTableTestUtils.java:82)
> 2021-05-19T21:28:02.8721311Z May 19 21:28:02 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.wordFreqToUpsertKafka(UpsertKafkaTableITCase.java:440)
> 2021-05-19T21:28:02.8730402Z May 19 21:28:02 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.testAggregate(UpsertKafkaTableITCase.java:73)
> 2021-05-19T21:28:02.8731390Z May 19 21:28:02 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-19T21:28:02.8732095Z May 19 21:28:02 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-19T21:28:02.8732935Z May 19 21:28:02 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-19T21:28:02.8733726Z May 19 21:28:02 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-19T21:28:02.8734598Z May 19 21:28:02 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-19T21:28:02.8735450Z May 19 21:28:02 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-19T21:28:02.8736313Z May 19 21:28:02 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-19T21:28:02.8737329Z May 19 21:28:02 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-19T21:28:02.8738165Z May 19 21:28:02 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-05-19T21:28:02.8738989Z May 19 21:28:02 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-05-19T21:28:02.8739741Z May 19 21:28:02 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-19T21:28:02.8740563Z May 19 21:28:02 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-19T21:28:02.8741340Z May 19 21:28:02 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-19T21:28:02.8742077Z May 19 21:28:02 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-19T21:28:02.8742802Z May 19 21:28:02 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-19T21:28:02.8743594Z May 19 21:28:02 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-19T21:28:02.8744811Z May 19 21:28:02 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>
[jira] [Commented] (FLINK-21979) Job can be restarted from the beginning after it reached a terminal state
[
https://issues.apache.org/jira/browse/FLINK-21979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348137#comment-17348137
]
Till Rohrmann commented on FLINK-21979:
---
We need to fix this problem in {{1.14.0}}.
> Job can be restarted from the beginning after it reached a terminal state
> -
>
> Key: FLINK-21979
> URL: https://issues.apache.org/jira/browse/FLINK-21979
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0
>
>
> Currently, the {{JobMaster}} removes all checkpoints after a job reaches a
> globally terminal state. Then it notifies the {{Dispatcher}} about the
> termination of the job. The {{Dispatcher}} then removes the job from the
> {{SubmittedJobGraphStore}}. If the {{Dispatcher}} process fails before doing
> that it might get restarted. In this case, the {{Dispatcher}} would still
> find the job in the {{SubmittedJobGraphStore}} and recover it. Since the
> {{CompletedCheckpointStore}} is empty, it would start executing this job from
> the beginning.
> I think we must not remove job state before the job has not been marked as
> done or made inaccessible for any restarted processes. Concretely, we should
> first remove the job from the {{SubmittedJobGraphStore}} and only then delete
> the checkpoints. Ideally all the job related cleanup operation happens
> atomically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21979) Job can be restarted from the beginning after it reached a terminal state
[
https://issues.apache.org/jira/browse/FLINK-21979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-21979:
--
Labels: (was: stale-critical)
> Job can be restarted from the beginning after it reached a terminal state
> -
>
> Key: FLINK-21979
> URL: https://issues.apache.org/jira/browse/FLINK-21979
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0
>
>
> Currently, the {{JobMaster}} removes all checkpoints after a job reaches a
> globally terminal state. Then it notifies the {{Dispatcher}} about the
> termination of the job. The {{Dispatcher}} then removes the job from the
> {{SubmittedJobGraphStore}}. If the {{Dispatcher}} process fails before doing
> that it might get restarted. In this case, the {{Dispatcher}} would still
> find the job in the {{SubmittedJobGraphStore}} and recover it. Since the
> {{CompletedCheckpointStore}} is empty, it would start executing this job from
> the beginning.
> I think we must not remove job state before the job has not been marked as
> done or made inaccessible for any restarted processes. Concretely, we should
> first remove the job from the {{SubmittedJobGraphStore}} and only then delete
> the checkpoints. Ideally all the job related cleanup operation happens
> atomically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20427) Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to data loss
[
https://issues.apache.org/jira/browse/FLINK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-20427:
--
Labels: (was: stale-critical)
> Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to
> data loss
> ---
>
> Key: FLINK-20427
> URL: https://issues.apache.org/jira/browse/FLINK-20427
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / Checkpointing
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0
>
>
> The {{CheckpointConfig.setPreferCheckpointForRecovery}} allows to configure
> whether Flink prefers checkpoints for recovery if the
> {{CompletedCheckpointStore}} contains savepoints and checkpoints. This is
> problematic because due to this feature, Flink might prefer older checkpoints
> over newer savepoints for recovery. Since some components expect that the
> always the latest checkpoint/savepoint is used (e.g. the
> {{SourceCoordinator}}), it breaks assumptions and can lead to
> {{SourceSplits}} which are not read. This effectively means that the system
> loses data. Similarly, this behaviour can cause that exactly once sinks might
> output results multiple times which violates the processing guarantees.
> Hence, I believe that we should remove this setting because it changes
> Flink's behaviour in some very significant way potentially w/o the user
> noticing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20427) Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to data loss
[
https://issues.apache.org/jira/browse/FLINK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348138#comment-17348138
]
Till Rohrmann commented on FLINK-20427:
---
I think this is still a very important issue to sort out and fix because it can
lead to unwanted data loss.
> Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to
> data loss
> ---
>
> Key: FLINK-20427
> URL: https://issues.apache.org/jira/browse/FLINK-20427
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / Checkpointing
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0
>
>
> The {{CheckpointConfig.setPreferCheckpointForRecovery}} allows to configure
> whether Flink prefers checkpoints for recovery if the
> {{CompletedCheckpointStore}} contains savepoints and checkpoints. This is
> problematic because due to this feature, Flink might prefer older checkpoints
> over newer savepoints for recovery. Since some components expect that the
> always the latest checkpoint/savepoint is used (e.g. the
> {{SourceCoordinator}}), it breaks assumptions and can lead to
> {{SourceSplits}} which are not read. This effectively means that the system
> loses data. Similarly, this behaviour can cause that exactly once sinks might
> output results multiple times which violates the processing guarantees.
> Hence, I believe that we should remove this setting because it changes
> Flink's behaviour in some very significant way potentially w/o the user
> noticing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21928) DuplicateJobSubmissionException after JobManager failover
[ https://issues.apache.org/jira/browse/FLINK-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-21928: -- Labels: (was: stale-critical) > DuplicateJobSubmissionException after JobManager failover > - > > Key: FLINK-21928 > URL: https://issues.apache.org/jira/browse/FLINK-21928 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.10.3, 1.11.3, 1.12.2, 1.13.0 > Environment: StandaloneApplicationClusterEntryPoint using a fixed job > ID, High Availability enabled >Reporter: Ufuk Celebi >Priority: Critical > Fix For: 1.14.0 > > > Consider the following scenario: > * Environment: StandaloneApplicationClusterEntryPoint using a fixed job ID, > high availability enabled > * Flink job reaches a globally terminal state > * Flink job is marked as finished in the high-availability service's > RunningJobsRegistry > * The JobManager fails over > On recovery, the [Dispatcher throws DuplicateJobSubmissionException, because > the job is marked as done in the > RunningJobsRegistry|https://github.com/apache/flink/blob/release-1.12.2/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L332-L340]. > When this happens, users cannot get out of the situation without manually > redeploying the JobManager process and changing the job ID^1^. > The desired semantics are that we don't want to re-execute a job that has > reached a globally terminal state. In this particular case, we know that the > job has already reached such a state (as it has been marked in the registry). > Therefore, we could handle this case by executing the regular termination > sequence instead of throwing a DuplicateJobSubmission. > --- > ^1^ With ZooKeeper HA, the respective node is not ephemeral. In Kubernetes > HA, there is no notion of ephemeral data that is tied to a session in the > first place afaik. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22437) Miss adding parallesim for filter operator in batch mode
[
https://issues.apache.org/jira/browse/FLINK-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348139#comment-17348139
]
godfrey he commented on FLINK-22437:
[~zoucao] yes, I think that is the easiest approach
> Miss adding parallesim for filter operator in batch mode
>
>
> Key: FLINK-22437
> URL: https://issues.apache.org/jira/browse/FLINK-22437
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.2
>Reporter: zoucao
>Priority: Major
>
> when I execute batch sql as follow in flink-1.12.2, I found lots of small
> files in hdfs. In filesystem connector, `GroupedPartitionWriter` will be
> used, and it close the last partiton if a new record does not belong to the
> existing partition. The phenomenon occurred if there are more than one
> partiton's records are sent to filesystem sink at the same time. Hive source
> can determine parallesim by the number of file and partition, and the
> parallesim will extended by sort operator, but in
> `CommonPhysicalSink#createSinkTransformation`，a filter operator will be add
> to support `SinkNotNullEnforcer`, there is no parallesim set for it, so
> filesystem sink operator can not get the correct parallesim from inputstream.
> {code:java}
> CREATE CATALOG myHive with (
> 'type'='hive',
> 'property-version'='1',
> 'default-database' = 'flink_sql_online_test'
> );
> -- SET table.sql-dialect=hive;
> -- CREATE TABLE IF NOT EXISTS myHive.flink_sql_online_test.hive_sink (
> --`timestamp` BIGINT,
> --`time` STRING,
> --id BIGINT,
> --product STRING,
> --price DOUBLE,
> --canSell STRING,
> --selledNum BIGINT
> -- ) PARTITIONED BY (
> --dt STRING,
> --`hour` STRING,
> -- `min` STRING
> -- ) TBLPROPERTIES (
> --'partition.time-extractor.timestamp-pattern'='$dt $hr:$min:00',
> --'sink.partition-commit.trigger'='partition-time',
> --'sink.partition-commit.delay'='1 min',
> --'sink.partition-commit.policy.kind'='metastore,success-file'
> -- );
> create table fs_sink (
> `timestamp` BIGINT,
> `time` STRING,
> id BIGINT,
> product STRING,
> price DOUBLE,
> canSell STRING,
> selledNum BIGINT,
> dt STRING,
> `hour` STRING,
> `min` STRING
> ) PARTITIONED BY (dt, `hour`, `min`) with (
> 'connector'='filesystem',
> 'path'='hdfs://',
> 'format'='csv'
> );
> insert into fs_sink
> select * from myHive.flink_sql_online_test.hive_sink;
> {code}
> I think this problem can be fixed by adding a parallesim for it just like
> {code:java}
> val dataStream = new DataStream(env, inputTransformation).filter(enforcer)
> .setParallelism(inputTransformation.getParallelism)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-22437) Miss adding parallesim for filter operator in batch mode
[
https://issues.apache.org/jira/browse/FLINK-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he reassigned FLINK-22437:
--
Assignee: zoucao
> Miss adding parallesim for filter operator in batch mode
>
>
> Key: FLINK-22437
> URL: https://issues.apache.org/jira/browse/FLINK-22437
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.2
>Reporter: zoucao
>Assignee: zoucao
>Priority: Major
>
> when I execute batch sql as follow in flink-1.12.2, I found lots of small
> files in hdfs. In filesystem connector, `GroupedPartitionWriter` will be
> used, and it close the last partiton if a new record does not belong to the
> existing partition. The phenomenon occurred if there are more than one
> partiton's records are sent to filesystem sink at the same time. Hive source
> can determine parallesim by the number of file and partition, and the
> parallesim will extended by sort operator, but in
> `CommonPhysicalSink#createSinkTransformation`，a filter operator will be add
> to support `SinkNotNullEnforcer`, there is no parallesim set for it, so
> filesystem sink operator can not get the correct parallesim from inputstream.
> {code:java}
> CREATE CATALOG myHive with (
> 'type'='hive',
> 'property-version'='1',
> 'default-database' = 'flink_sql_online_test'
> );
> -- SET table.sql-dialect=hive;
> -- CREATE TABLE IF NOT EXISTS myHive.flink_sql_online_test.hive_sink (
> --`timestamp` BIGINT,
> --`time` STRING,
> --id BIGINT,
> --product STRING,
> --price DOUBLE,
> --canSell STRING,
> --selledNum BIGINT
> -- ) PARTITIONED BY (
> --dt STRING,
> --`hour` STRING,
> -- `min` STRING
> -- ) TBLPROPERTIES (
> --'partition.time-extractor.timestamp-pattern'='$dt $hr:$min:00',
> --'sink.partition-commit.trigger'='partition-time',
> --'sink.partition-commit.delay'='1 min',
> --'sink.partition-commit.policy.kind'='metastore,success-file'
> -- );
> create table fs_sink (
> `timestamp` BIGINT,
> `time` STRING,
> id BIGINT,
> product STRING,
> price DOUBLE,
> canSell STRING,
> selledNum BIGINT,
> dt STRING,
> `hour` STRING,
> `min` STRING
> ) PARTITIONED BY (dt, `hour`, `min`) with (
> 'connector'='filesystem',
> 'path'='hdfs://',
> 'format'='csv'
> );
> insert into fs_sink
> select * from myHive.flink_sql_online_test.hive_sink;
> {code}
> I think this problem can be fixed by adding a parallesim for it just like
> {code:java}
> val dataStream = new DataStream(env, inputTransformation).filter(enforcer)
> .setParallelism(inputTransformation.getParallelism)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22437) Miss adding parallesim for filter operator in batch mode
[
https://issues.apache.org/jira/browse/FLINK-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348139#comment-17348139
]
godfrey he edited comment on FLINK-22437 at 5/20/21, 8:03 AM:
--
[~zoucao] yes, I think that is the easiest approach. Assign to you
was (Author: godfreyhe):
[~zoucao] yes, I think that is the easiest approach
> Miss adding parallesim for filter operator in batch mode
>
>
> Key: FLINK-22437
> URL: https://issues.apache.org/jira/browse/FLINK-22437
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.2
>Reporter: zoucao
>Assignee: zoucao
>Priority: Major
>
> when I execute batch sql as follow in flink-1.12.2, I found lots of small
> files in hdfs. In filesystem connector, `GroupedPartitionWriter` will be
> used, and it close the last partiton if a new record does not belong to the
> existing partition. The phenomenon occurred if there are more than one
> partiton's records are sent to filesystem sink at the same time. Hive source
> can determine parallesim by the number of file and partition, and the
> parallesim will extended by sort operator, but in
> `CommonPhysicalSink#createSinkTransformation`，a filter operator will be add
> to support `SinkNotNullEnforcer`, there is no parallesim set for it, so
> filesystem sink operator can not get the correct parallesim from inputstream.
> {code:java}
> CREATE CATALOG myHive with (
> 'type'='hive',
> 'property-version'='1',
> 'default-database' = 'flink_sql_online_test'
> );
> -- SET table.sql-dialect=hive;
> -- CREATE TABLE IF NOT EXISTS myHive.flink_sql_online_test.hive_sink (
> --`timestamp` BIGINT,
> --`time` STRING,
> --id BIGINT,
> --product STRING,
> --price DOUBLE,
> --canSell STRING,
> --selledNum BIGINT
> -- ) PARTITIONED BY (
> --dt STRING,
> --`hour` STRING,
> -- `min` STRING
> -- ) TBLPROPERTIES (
> --'partition.time-extractor.timestamp-pattern'='$dt $hr:$min:00',
> --'sink.partition-commit.trigger'='partition-time',
> --'sink.partition-commit.delay'='1 min',
> --'sink.partition-commit.policy.kind'='metastore,success-file'
> -- );
> create table fs_sink (
> `timestamp` BIGINT,
> `time` STRING,
> id BIGINT,
> product STRING,
> price DOUBLE,
> canSell STRING,
> selledNum BIGINT,
> dt STRING,
> `hour` STRING,
> `min` STRING
> ) PARTITIONED BY (dt, `hour`, `min`) with (
> 'connector'='filesystem',
> 'path'='hdfs://',
> 'format'='csv'
> );
> insert into fs_sink
> select * from myHive.flink_sql_online_test.hive_sink;
> {code}
> I think this problem can be fixed by adding a parallesim for it just like
> {code:java}
> val dataStream = new DataStream(env, inputTransformation).filter(enforcer)
> .setParallelism(inputTransformation.getParallelism)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-22686) Incompatible subtask mappings while resuming from unaligned checkpoints
[
https://issues.apache.org/jira/browse/FLINK-22686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz reassigned FLINK-22686:
Assignee: Dawid Wysakowicz
> Incompatible subtask mappings while resuming from unaligned checkpoints
> ---
>
> Key: FLINK-22686
> URL: https://issues.apache.org/jira/browse/FLINK-22686
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Arvid Heise
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Labels: stale-blocker
> Fix For: 1.13.1
>
> Attachments: topology_1.png, topology_2.png, topology_3.png
>
>
> A user
> [reported|https://lists.apache.org/x/list.html?u...@flink.apache.org:lte=1M:Flink%201.13.0%20reactive%20mode:%20Job%20stop%20and%20cannot%20restore%20from%20checkpoint]
> that he encountered an internal error while resuming during reactive mode.
> There isn't an immediate connection to reactive mode, so it's safe to assume
> that one rescaling case was not covered.
> {noformat}
> Caused by: java.lang.IllegalStateException: Incompatible subtask mappings:
> are multiple operators ingesting/producing intermediate results with varying
> degrees of parallelism?Found RescaleMappings{mappings=[[0, 1, 2, 3, 4, 5, 6,
> 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
> 27, 28, 29], [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
> 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59], [60, 61, 62, 63, 64,
> 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
> 84, 85, 86, 87, 88, 89], [90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101,
> 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
> 117, 118, 119], [120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131,
> 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146,
> 147, 148, 149], [150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161,
> 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176,
> 177, 178, 179], [180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
> 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206,
> 207, 208, 209]]} and RescaleMappings{mappings=[[0, 7, 14, 21, 28, 35, 42, 49,
> 56, 63, 70, 77, 84, 91, 98, 105, 112, 119, 126, 133, 140, 147, 154, 161, 168,
> 175, 182, 189, 196, 203], [1, 8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85,
> 92, 99, 106, 113, 120, 127, 134, 141, 148, 155, 162, 169, 176, 183, 190, 197,
> 204], [2, 9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86, 93, 100, 107, 114,
> 121, 128, 135, 142, 149, 156, 163, 170, 177, 184, 191, 198, 205], [3, 10, 17,
> 24, 31, 38, 45, 52, 59, 66, 73, 80, 87, 94, 101, 108, 115, 122, 129, 136,
> 143, 150, 157, 164, 171, 178, 185, 192, 199, 206], [4, 11, 18, 25, 32, 39,
> 46, 53, 60, 67, 74, 81, 88, 95, 102, 109, 116, 123, 130, 137, 144, 151, 158,
> 165, 172, 179, 186, 193, 200, 207], [5, 12, 19, 26, 33, 40, 47, 54, 61, 68,
> 75, 82, 89, 96, 103, 110, 117, 124, 131, 138, 145, 152, 159, 166, 173, 180,
> 187, 194, 201, 208], [6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90, 97,
> 104, 111, 118, 125, 132, 139, 146, 153, 160, 167, 174, 181, 188, 195, 202,
> 209]]}.
> at
> org.apache.flink.runtime.checkpoint.TaskStateAssignment.checkSubtaskMapping(TaskStateAssignment.java:322)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.checkpoint.TaskStateAssignment.getInputMapping(TaskStateAssignment.java:306)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.reDistributeInputChannelStates(StateAssignmentOperation.java:409)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignAttemptState(StateAssignmentOperation.java:193)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignStates(StateAssignmentOperation.java:139)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedStateInternal(CheckpointCoordinator.java:1566)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1646)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.tryRestoreExecutionGraphFromSavepoint(DefaultExecutionGraphFactory.java:163)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreEx
[GitHub] [flink] XComp commented on pull request #15945: [FLINK-22688][runtime] Eases assertion on ExceptionHistoryEntry
XComp commented on pull request #15945: URL: https://github.com/apache/flink/pull/15945#issuecomment-844846086 We had this problem come up in another thread of @Airblader : The failure that might have also caused the issue in FLINK-22688 is that the Task gets scheduled but the TaskManager does not confirm the slot allocation due to network issues or the TaskManager becoming unavailable. We run into a slot request timeout triggering a `NoResourceAvailableException` for the job. The `TaskManagerLocation` was not set in that case, yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22462) JdbcExactlyOnceSinkE2eTest.testInsert failed because of too many clients.
[
https://issues.apache.org/jira/browse/FLINK-22462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348142#comment-17348142
]
Till Rohrmann commented on FLINK-22462:
---
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18130&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=03dca39c-73e8-5aaf-601d-328ae5c35f20&l=13457
> JdbcExactlyOnceSinkE2eTest.testInsert failed because of too many clients.
> -
>
> Key: FLINK-22462
> URL: https://issues.apache.org/jira/browse/FLINK-22462
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.14.0
>Reporter: Guowei Ma
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17178&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=bfbc6239-57a0-5db0-63f3-41551b4f7d51&l=13514
> {code:java}
> Apr 25 23:05:31 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 138.743 s <<< FAILURE! - in
> org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest
> Apr 25 23:05:31 [ERROR]
> testInsert(org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest)
> Time elapsed: 137.267 s <<< ERROR!
> Apr 25 23:05:31 org.postgresql.util.PSQLException: FATAL: sorry, too many
> clients already
> Apr 25 23:05:31 at
> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:524)
> Apr 25 23:05:31 at
> org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:145)
> Apr 25 23:05:31 at
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:196)
> Apr 25 23:05:31 at
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
> Apr 25 23:05:31 at
> org.postgresql.jdbc.PgConnection.(PgConnection.java:211)
> Apr 25 23:05:31 at org.postgresql.Driver.makeConnection(Driver.java:459)
> Apr 25 23:05:31 at org.postgresql.Driver.connect(Driver.java:261)
> Apr 25 23:05:31 at
> java.sql.DriverManager.getConnection(DriverManager.java:664)
> Apr 25 23:05:31 at
> java.sql.DriverManager.getConnection(DriverManager.java:247)
> Apr 25 23:05:31 at
> org.apache.flink.connector.jdbc.xa.JdbcXaFacadeTestHelper.getInsertedIds(JdbcXaFacadeTestHelper.java:81)
> Apr 25 23:05:31 at
> org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest.testInsert(JdbcExactlyOnceSinkE2eTest.java:119)
> Apr 25 23:05:31 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann closed pull request #15954: [FLINK-22704][tests] Harden ZooKeeperHaServicesTest.testCleanupJobData
tillrohrmann closed pull request #15954: URL: https://github.com/apache/flink/pull/15954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #15955: [BP-1.13][FLINK-22704][tests] Harden ZooKeeperHaServicesTest.testCleanupJobData
tillrohrmann commented on pull request #15955: URL: https://github.com/apache/flink/pull/15955#issuecomment-844848313 Thanks for the review @XComp. Merging this PR now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan merged pull request #15960: [FLINK-22692][tests] Disable CheckpointStoreITCase with adaptive scheduler
rkhachatryan merged pull request #15960: URL: https://github.com/apache/flink/pull/15960 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann merged pull request #15955: [BP-1.13][FLINK-22704][tests] Harden ZooKeeperHaServicesTest.testCleanupJobData
tillrohrmann merged pull request #15955: URL: https://github.com/apache/flink/pull/15955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann merged pull request #15956: [BP-1.12][FLINK-22704][tests] Harden ZooKeeperHaServicesTest.testCleanupJobData
tillrohrmann merged pull request #15956: URL: https://github.com/apache/flink/pull/15956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-22704) ZooKeeperHaServicesTest.testCleanupJobData failed
[
https://issues.apache.org/jira/browse/FLINK-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-22704.
-
Fix Version/s: 1.12.5
1.13.1
1.14.0
Resolution: Fixed
Fixed via
1.14.0: 3558ff172c73fad95cb08bb2e6cda96daf13db91
1.13.1: 7390981e088e618af91a4493d720a36572e8672c
1.12.5: 7b9351e26d2356ac1d2f4f1ad02fd7f6860257be
> ZooKeeperHaServicesTest.testCleanupJobData failed
> -
>
> Key: FLINK-22704
> URL: https://issues.apache.org/jira/browse/FLINK-22704
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.1, 1.12.5
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18108&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=7c61167f-30b3-5893-cc38-a9e3d057e392&l=8172
> {code:java}
> May 19 01:30:02 Expected: a collection containing
> "1a2850d5759a2e1f4fef9cc7e6abc675"
> May 19 01:30:02 but: was "resource_manager_lock"
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-22661) HiveInputFormatPartitionReader can return invalid data
[ https://issues.apache.org/jira/browse/FLINK-22661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-22661. Fix Version/s: 1.13.1 1.14.0 Resolution: Fixed Fixed in master: 74f4b7fc61a8222716723f308119da287a3af953 Fixed in release-1.13: c88c0c98198b061e351d7574a5c5b891ba38116b > HiveInputFormatPartitionReader can return invalid data > -- > > Key: FLINK-22661 > URL: https://issues.apache.org/jira/browse/FLINK-22661 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.1 > > > This happens when there're multiple splits to read, and an extra record is > returned when we switch splits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rkhachatryan opened a new pull request #15967: [FLINK-22692][tests] Disable CheckpointStoreITCase with adaptive scheduler
rkhachatryan opened a new pull request #15967: URL: https://github.com/apache/flink/pull/15967 Backport of #15960 to 1.12 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan opened a new pull request #15968: [FLINK-22692][tests] Disable CheckpointStoreITCase with adaptive scheduler
rkhachatryan opened a new pull request #15968: URL: https://github.com/apache/flink/pull/15968 Backport of #15960 to 1.13 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15004: [FLINK-21253][table-planner-blink] Support grouping set syntax for GroupWindowAggregate
flinkbot edited a comment on pull request #15004: URL: https://github.com/apache/flink/pull/15004#issuecomment-785052726 ## CI report: * b826a818fcd0d47f2454575c59d324519e2c6ad9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18140) * 4b01a38ffe4968511607b84c58309bf75183085a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18168) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15967: [FLINK-22692][tests] Disable CheckpointStoreITCase with adaptive scheduler
flinkbot commented on pull request #15967: URL: https://github.com/apache/flink/pull/15967#issuecomment-844854982 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5d4526fdef21ee80d26be789a65c894f957d2517 (Thu May 20 08:24:44 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15968: [FLINK-22692][tests] Disable CheckpointStoreITCase with adaptive scheduler
flinkbot commented on pull request #15968: URL: https://github.com/apache/flink/pull/15968#issuecomment-844854950 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b6185c6da599d4645933695a74ec897549b5a6af (Thu May 20 08:24:40 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15965: [datagen] Normalize parameter names in RandomGeneratorVisitor and SequenceGeneratorVisitor
flinkbot edited a comment on pull request #15965: URL: https://github.com/apache/flink/pull/15965#issuecomment-844823334 ## CI report: * 9ab417c63ef4d6a0f4f3b785980914395901e955 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18170) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15966: [FLINK-22659][docs] Add execution.checkpointing.interval to configura…
flinkbot edited a comment on pull request #15966: URL: https://github.com/apache/flink/pull/15966#issuecomment-844823503 ## CI report: * 50986672248961509dc127d1df42ec8ef96b5ecd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18171) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15964: [BP-1.13][FLINK-22434] Store suspended execution graphs on termination to keep them accessible
flinkbot edited a comment on pull request #15964: URL: https://github.com/apache/flink/pull/15964#issuecomment-844823190 ## CI report: * c2550460db2c0951911dd3219190d0f59e9d1ac6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18169) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22721) Breaking HighAvailabilityServices interface by adding new method
Till Rohrmann created FLINK-22721:
-
Summary: Breaking HighAvailabilityServices interface by adding new
method
Key: FLINK-22721
URL: https://issues.apache.org/jira/browse/FLINK-22721
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.14.0, 1.13.1, 1.12.5
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.14.0, 1.13.1, 1.12.5
As part of FLINK-20695 we introduced a new method to the
{{HighAvailabilityServices.cleanupJobData}} interface. Since this method has
not default implementation it is currently breaking change. Since Flink allows
to implement custom Ha services using this interface, I suggest adding a
default implementation for this method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22088) CheckpointCoordinator might not be able to abort triggering checkpoint if failover happens during triggering
[
https://issues.apache.org/jira/browse/FLINK-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348149#comment-17348149
]
Yun Gao commented on FLINK-22088:
-
Hi [~pnowojski], very sorry for missing the notification and reply late. I
think the fix is not too difficult: we would need to ensure the global state
check is in the same lock area with recording pending checkpoint, then we could
ensure either the checkpoint is not recorded after job failed, or the
checkpoint is recorded before job failed and job failure could abort it.
Since we must call _statTracker.report()_ complete before we exit the lock area
that adds pending checkpoint into the _pendingCheckpoints_ (otherwise if the
pending checkpoint is aborted after we left the lock, it would try to report
fail status, if we have not track the pending checkpoint yet, there would be
errors), thus there might be two options:
* Move the whole process into the lock area:
{code:java}
synchronized(lock) {
try {
check global state;
} catch (Exception e) {
throw e;
}
pendingCheckpoint = new PendingCheckpoint();
trackPendingCheckpoints(pendingCheckpoint);
}
{code}
* Move the check into the lock area, after the pending checkpoint get created
and tracked.
{code:java}
pendingCheckpoint = new PendingCheckpoint();
trackPendingCheckpoints(pendingCheckpoint);
synchronized(lock) {
try {
check global state;
} catch (Exception) {
pendingCheckpoint.abort(...);
throw e;
}
}
{code}
I tend to the second option to not increase the time in the lock area.
> CheckpointCoordinator might not be able to abort triggering checkpoint if
> failover happens during triggering
>
>
> Key: FLINK-22088
> URL: https://issues.apache.org/jira/browse/FLINK-22088
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Yun Gao
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently when job failover, it would try to cancel all the pending
> checkpoint via CheckpointCoordinatorDeActivator#jobStatusChanges ->
> stopCheckpointScheduler, it would try to cancel all the pending checkpoints
> and also set periodicScheduling to false.
> If at this time there is just one checkpoint start triggering, it might
> acquire all the execution to trigger before failover and start triggering.
> ideally it should be aborted in createPendingCheckpoint->
> preCheckGlobalState. However, since the check and creating pending checkpoint
> is in two different scope, there might be cases the
> CheckpointCoordinator#stopCheckpointScheduler happens during the two lock
> scope.
> We may optimize this checking; However, since the execution would finally
> fail to trigger checkpoint, it should not affect the rightness of the job.
> Besides, even if we optimize it, there might still be cases that the
> execution trigger failed due to concurrent failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22685) Write data to hive table in batch mode throws FileNotFoundException.
[
https://issues.apache.org/jira/browse/FLINK-22685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348151#comment-17348151
]
Rui Li commented on FLINK-22685:
Hi [~ZhangChaoming], the staging folder is created on client side, and we want
to clean it up in case of job compiling/submission failures. So I guess you
need to wait for the job to finish. You can set {{table.dml-sync=true}} to make
SQL client run DMLs synchronously.
> Write data to hive table in batch mode throws FileNotFoundException.
>
>
> Key: FLINK-22685
> URL: https://issues.apache.org/jira/browse/FLINK-22685
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Environment: Flink Based on Flink 1.11.1.
>Reporter: Moses
>Priority: Minor
>
> h3. Scence
> I wanna luanch a batch job to process hive table data and write the result to
> another table(*T1*), and my SQL statements is wriiten like below:
> {code:sql}
> -- use hive dialect
> SET table.sql-dialect=hive;
> -- insert into hive table
> insert overwrite table T1
> partition (p_day_id,p_file_id)
> select distinct
> {code}
> The job was success luanched, but it failed on *Sink* operator. On Flink UI
> page I saw all task state is `*FINISHED*`, but *the job failed and it
> restarted again*.
> And I found exception information like below: (*The path was marksed*)
> {code:java}
> java.lang.Exception: Failed to finalize execution on master
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1291)
> at
> org.apache.flink.runtime.executiongraph.ExecutionVertex.executionFinished(ExecutionVertex.java:870)
> at
> org.apache.flink.runtime.executiongraph.Execution.markFinished(Execution.java:1125)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.updateStateInternal(ExecutionGraph.java:1491)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.updateState(ExecutionGraph.java:1464)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:497)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:386)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: org.apache.flink.table.api.TableException: Exception in
> finalizeGlobal
> at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:97)
> at
> org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:132)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1286)
> ... 31 more
> Caused by: java.io.FileNotFoundException: File
> //XX/XXX/.staging_1621244168369 does not exist.
> at
> org.apache.hadoop.hdfs.Distrib
[jira] [Commented] (FLINK-22721) Breaking HighAvailabilityServices interface by adding new method
[
https://issues.apache.org/jira/browse/FLINK-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348154#comment-17348154
]
Chesnay Schepler commented on FLINK-22721:
--
What exact guarantees do we provide for this interface? It's not tagged as an
API (so maybe let's add {{@PublicEvolving}}?) and it exposes various various
other interfaces (BlobStore, JobGraphStore, LeaderRetrievalService, and further
down even the JobGraph); are we just maintaining compatibility on a best-effort
basis?
> Breaking HighAvailabilityServices interface by adding new method
>
>
> Key: FLINK-22721
> URL: https://issues.apache.org/jira/browse/FLINK-22721
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.13.1, 1.12.5
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0, 1.13.1, 1.12.5
>
>
> As part of FLINK-20695 we introduced a new method to the
> {{HighAvailabilityServices.cleanupJobData}} interface. Since this method has
> not default implementation it is currently breaking change. Since Flink
> allows to implement custom Ha services using this interface, I suggest adding
> a default implementation for this method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22721) Breaking HighAvailabilityServices interface by adding new method
[
https://issues.apache.org/jira/browse/FLINK-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348155#comment-17348155
]
Till Rohrmann commented on FLINK-22721:
---
The short answer is yes. We noticed the breaking behavior with the vvp-ha
service implementation. I think it is worthwhile trying to be as compatible as
possible since the default method won't hurt us much.
> Breaking HighAvailabilityServices interface by adding new method
>
>
> Key: FLINK-22721
> URL: https://issues.apache.org/jira/browse/FLINK-22721
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.13.1, 1.12.5
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0, 1.13.1, 1.12.5
>
>
> As part of FLINK-20695 we introduced a new method to the
> {{HighAvailabilityServices.cleanupJobData}} interface. Since this method has
> not default implementation it is currently breaking change. Since Flink
> allows to implement custom Ha services using this interface, I suggest adding
> a default implementation for this method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan closed pull request #15967: [BP-1.12][FLINK-22692][tests] Disable CheckpointStoreITCase with adaptive scheduler
rkhachatryan closed pull request #15967: URL: https://github.com/apache/flink/pull/15967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22692) CheckpointStoreITCase.testRestartOnRecoveryFailure fails with RuntimeException
[
https://issues.apache.org/jira/browse/FLINK-22692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-22692:
--
Fix Version/s: (was: 1.12.5)
> CheckpointStoreITCase.testRestartOnRecoveryFailure fails with RuntimeException
> --
>
> Key: FLINK-22692
> URL: https://issues.apache.org/jira/browse/FLINK-22692
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Robert Metzger
>Assignee: Roman Khachatryan
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.1
>
>
> Not sure if it is related to the adaptive scheduler:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18052&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=a0a633b8-47ef-5c5a-2806-3c13b9e48228
> {code}
> May 17 22:29:11 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 1.351 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.CheckpointStoreITCase
> May 17 22:29:11 [ERROR]
> testRestartOnRecoveryFailure(org.apache.flink.test.checkpointing.CheckpointStoreITCase)
> Time elapsed: 1.138 s <<< ERROR!
> May 17 22:29:11 org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> May 17 22:29:11 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> May 17 22:29:11 at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> May 17 22:29:11 at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> May 17 22:29:11 at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
> May 17 22:29:11 at akka.dispatch.OnComplete.internal(Future.scala:264)
> May 17 22:29:11 at akka.dispatch.OnComplete.internal(Future.scala:261)
> May 17 22:29:11 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> May 17 22:29:11 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> May 17 22:29:11 at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> May 17 22:29:11 at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> May 17 22:29:11 at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> May 17 22:29:11 at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> May 17 22:29:11 at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> May 17 22:29:11 at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann opened a new pull request #15969: [FLINK-22721][ha] Add default implementation for HighAvailabilityServices.cleanupJobData
tillrohrmann opened a new pull request #15969: URL: https://github.com/apache/flink/pull/15969 In order to not break the HighAvailabilityServices interface, this commit adds a default implementation for the HighAavilabilityServices.cleanupJobData method which does nothing. Users of this interface are recommended to override this method and add a specific implementation in order to clean up job specific HA services after a job completion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21952) Make all the "Connection reset by peer" exception wrapped as RemoteTransportException
[
https://issues.apache.org/jira/browse/FLINK-21952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348156#comment-17348156
]
Yun Gao commented on FLINK-21952:
-
I'll open a small PR for this issue (If there are no other problem).
> Make all the "Connection reset by peer" exception wrapped as
> RemoteTransportException
> -
>
> Key: FLINK-21952
> URL: https://issues.apache.org/jira/browse/FLINK-21952
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Reporter: Yun Gao
>Priority: Major
> Labels: stale-major
>
> In CreditBasedPartitionRequestClientHandler#exceptionCaught, the IOException
> or the exception with exact message "Connection reset by peer" are marked as
> RemoteTransportException.
> However, with the current Netty implementation, sometimes it might throw
> {code:java}
> org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer
> {code}
> in some case. It would be also wrapped as LocalTransportException, which
> might cause some confusion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15768: [FLINK-22451][table] Support (*) as parameter of UDFs in Table API
flinkbot edited a comment on pull request #15768: URL: https://github.com/apache/flink/pull/15768#issuecomment-826735938 ## CI report: * 4f398eeff8439c9c4c052c157cb5360a7b784d8c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18161) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22721) Breaking HighAvailabilityServices interface by adding new method
[
https://issues.apache.org/jira/browse/FLINK-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-22721:
---
Labels: pull-request-available (was: )
> Breaking HighAvailabilityServices interface by adding new method
>
>
> Key: FLINK-22721
> URL: https://issues.apache.org/jira/browse/FLINK-22721
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.13.1, 1.12.5
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.1, 1.12.5
>
>
> As part of FLINK-20695 we introduced a new method to the
> {{HighAvailabilityServices.cleanupJobData}} interface. Since this method has
> not default implementation it is currently breaking change. Since Flink
> allows to implement custom Ha services using this interface, I suggest adding
> a default implementation for this method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22692) CheckpointStoreITCase.testRestartOnRecoveryFailure fails with RuntimeException
[
https://issues.apache.org/jira/browse/FLINK-22692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-22692:
--
Affects Version/s: (was: 1.12.4)
> CheckpointStoreITCase.testRestartOnRecoveryFailure fails with RuntimeException
> --
>
> Key: FLINK-22692
> URL: https://issues.apache.org/jira/browse/FLINK-22692
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Robert Metzger
>Assignee: Roman Khachatryan
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.1, 1.12.5
>
>
> Not sure if it is related to the adaptive scheduler:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18052&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=a0a633b8-47ef-5c5a-2806-3c13b9e48228
> {code}
> May 17 22:29:11 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 1.351 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.CheckpointStoreITCase
> May 17 22:29:11 [ERROR]
> testRestartOnRecoveryFailure(org.apache.flink.test.checkpointing.CheckpointStoreITCase)
> Time elapsed: 1.138 s <<< ERROR!
> May 17 22:29:11 org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> May 17 22:29:11 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> May 17 22:29:11 at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> May 17 22:29:11 at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> May 17 22:29:11 at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> May 17 22:29:11 at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
> May 17 22:29:11 at akka.dispatch.OnComplete.internal(Future.scala:264)
> May 17 22:29:11 at akka.dispatch.OnComplete.internal(Future.scala:261)
> May 17 22:29:11 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> May 17 22:29:11 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> May 17 22:29:11 at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> May 17 22:29:11 at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> May 17 22:29:11 at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> May 17 22:29:11 at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> May 17 22:29:11 at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> May 17 22:29:11 at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol commented on a change in pull request #15927: [FLINK-22639][runtime] ClassLoaderUtil cannot print classpath of Flin…
zentol commented on a change in pull request #15927:
URL: https://github.com/apache/flink/pull/15927#discussion_r635900027
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoadersTest.java
##
@@ -208,6 +212,50 @@ public void testRepeatedParentFirstPatternClass() throws
Exception {
childClassLoader.close();
}
+@Test
+public void testGetClassLoaderInfo() throws Exception {
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+String formattedURL = ClassLoaderUtil.formatURL(childCodePath);
+
+assertEquals(
+ClassLoaderUtil.getUserCodeClassLoaderInfo(childClassLoader),
+"URL ClassLoader:" + formattedURL);
+
+childClassLoader.close();
+}
+
+@Test
+public void testGetClassLoaderInfoWithClassLoaderClosed() throws Exception
{
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+childClassLoader.close();
+
+assertThat(
+ClassLoaderUtil.getUserCodeClassLoaderInfo(childClassLoader),
+is(startsWith("Cannot access classloader info due to an
exception.")));
Review comment:
```suggestion
startsWith("Cannot access classloader info due to an
exception."));
```
I would think that `is` isn't necessary.
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoadersTest.java
##
@@ -208,6 +212,50 @@ public void testRepeatedParentFirstPatternClass() throws
Exception {
childClassLoader.close();
}
+@Test
+public void testGetClassLoaderInfo() throws Exception {
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+String formattedURL = ClassLoaderUtil.formatURL(childCodePath);
+
+assertEquals(
+ClassLoaderUtil.getUserCodeClassLoaderInfo(childClassLoader),
+"URL ClassLoader:" + formattedURL);
+
+childClassLoader.close();
Review comment:
if you want to ensure that this CL is closed then you should use a
finally block to ensure it is closed if the test fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann opened a new pull request #15970: [BP-1.13][FLINK-22721][ha] Add default implementation for HighAvailabilityServices.cleanupJobData
tillrohrmann opened a new pull request #15970: URL: https://github.com/apache/flink/pull/15970 Backport of #15969 to `release-1.13`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
flinkbot edited a comment on pull request #15924: URL: https://github.com/apache/flink/pull/15924#issuecomment-841943851 ## CI report: * 1685626df39ea331106f6b3f7d4f63f506a17f41 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18162) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15959: [FLINK-22684][runtime] Added ability to ignore in-flight data during …
flinkbot edited a comment on pull request #15959: URL: https://github.com/apache/flink/pull/15959#issuecomment-844265660 ## CI report: * f391dbf543590b9f0932b739ea697a2e7520aaea Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18147) * 4518db57c8fe60bcb46077ecdb4c8403045235a2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann opened a new pull request #15971: [BP-1.12][FLINK-22721][ha] Add default implementation for HighAvailabilityServices.cleanupJobData
tillrohrmann opened a new pull request #15971: URL: https://github.com/apache/flink/pull/15971 Backport of #15969 to `release-1.12`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15968: [BP-1.13][FLINK-22692][tests] Disable CheckpointStoreITCase with adaptive scheduler
flinkbot commented on pull request #15968: URL: https://github.com/apache/flink/pull/15968#issuecomment-844873946 ## CI report: * b6185c6da599d4645933695a74ec897549b5a6af UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-22529) StateFun Kinesis ingresses should support configs that are available via FlinkKinesisConsumer's ConsumerConfigConstants
[
https://issues.apache.org/jira/browse/FLINK-22529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igal Shilman reassigned FLINK-22529:
Assignee: Igal Shilman
> StateFun Kinesis ingresses should support configs that are available via
> FlinkKinesisConsumer's ConsumerConfigConstants
> ---
>
> Key: FLINK-22529
> URL: https://issues.apache.org/jira/browse/FLINK-22529
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Igal Shilman
>Priority: Major
> Fix For: statefun-3.1.0
>
>
> The Kinesis ingress should support the configs that are available in
> {{FlinkKinesisConsumer}}'s {{ConsumerConfigConstants}}. Instead, currently,
> all property keys provided to the Kinesis ingress are assumed to be
> AWS-client related keys, and therefore have all been appended with the
> `aws.clientconfigs` string.
> I'd suggest to avoid mixing the {{ConsumerConfigConstants}} configs within
> the properties as well. Having named methods on the {{KinesisIngressBuilder}}
> for those configuration would provide a cleaner solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-22529) StateFun Kinesis ingresses should support configs that are available via FlinkKinesisConsumer's ConsumerConfigConstants
[
https://issues.apache.org/jira/browse/FLINK-22529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igal Shilman reassigned FLINK-22529:
Assignee: Tzu-Li (Gordon) Tai (was: Igal Shilman)
> StateFun Kinesis ingresses should support configs that are available via
> FlinkKinesisConsumer's ConsumerConfigConstants
> ---
>
> Key: FLINK-22529
> URL: https://issues.apache.org/jira/browse/FLINK-22529
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Fix For: statefun-3.1.0
>
>
> The Kinesis ingress should support the configs that are available in
> {{FlinkKinesisConsumer}}'s {{ConsumerConfigConstants}}. Instead, currently,
> all property keys provided to the Kinesis ingress are assumed to be
> AWS-client related keys, and therefore have all been appended with the
> `aws.clientconfigs` string.
> I'd suggest to avoid mixing the {{ConsumerConfigConstants}} configs within
> the properties as well. Having named methods on the {{KinesisIngressBuilder}}
> for those configuration would provide a cleaner solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22531) Improve the support for finite streaming jobs with async operations.
[ https://issues.apache.org/jira/browse/FLINK-22531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igal Shilman updated FLINK-22531: - Fix Version/s: (was: statefun-3.1.0) > Improve the support for finite streaming jobs with async operations. > > > Key: FLINK-22531 > URL: https://issues.apache.org/jira/browse/FLINK-22531 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.0.0 >Reporter: Igal Shilman >Priority: Major > Labels: developer-experience > > Finite streaming jobs will terminate even in the presence of asynchronous in > flight operations. > Looking at the AsyncWait operator, it seems that it can be mitigated by using > the following interface: > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/BoundedOneInput.java#L27] > > Mailing list thread: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Stateful-Functions-Help-for-calling-remote-stateful-function-written-in-Python-td43283.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15969: [FLINK-22721][ha] Add default implementation for HighAvailabilityServices.cleanupJobData
flinkbot commented on pull request #15969: URL: https://github.com/apache/flink/pull/15969#issuecomment-844874688 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7baeca3e681bd57e9f9909b741b17148f2b81120 (Thu May 20 08:49:02 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #15961: [FLINK-22706][release]Update License information in NOTICE file
zentol commented on pull request #15961: URL: https://github.com/apache/flink/pull/15961#issuecomment-844876207 > The NOTICE file doesn't contain these information. Could you explain a bit? The NOTICE file in the root of the project is only for the source release. It only covers files are that actually contained in that releases, i.e., the contents of the repository with various exceptions. NOTICE files in generally only cover what is included in an artifact; flink-kubernetes contains a NOTICE file that covers all dependencies that we bundle in the flink-kubernetes jar; dependencies that aren't bundled do not need to be listed in the NOTICE. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15971: [BP-1.12][FLINK-22721][ha] Add default implementation for HighAvailabilityServices.cleanupJobData
flinkbot commented on pull request #15971: URL: https://github.com/apache/flink/pull/15971#issuecomment-844876461 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit da9a525ef82916467de4c8b59e6f0a34ac75a191 (Thu May 20 08:51:06 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15970: [BP-1.13][FLINK-22721][ha] Add default implementation for HighAvailabilityServices.cleanupJobData
flinkbot commented on pull request #15970: URL: https://github.com/apache/flink/pull/15970#issuecomment-844876488 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 852aa78981e7a423b2a116bff0b4a16fe9478b56 (Thu May 20 08:51:09 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on pull request #15966: [FLINK-22659][docs] Add execution.checkpointing.interval to configura…
Myasuka commented on pull request #15966: URL: https://github.com/apache/flink/pull/15966#issuecomment-844876525 @lixmgl Why not resolve the conflicts in your PR as hinted below?  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-22389) Expose Message Batches to SDKs
[ https://issues.apache.org/jira/browse/FLINK-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igal Shilman reassigned FLINK-22389: Assignee: Igal Shilman > Expose Message Batches to SDKs > -- > > Key: FLINK-22389 > URL: https://issues.apache.org/jira/browse/FLINK-22389 > Project: Flink > Issue Type: New Feature > Components: Stateful Functions >Reporter: Konstantin Knauf >Assignee: Igal Shilman >Priority: Major > Fix For: statefun-3.1.0 > > > Stateful Functions always invokes a functions with a batch of messages. Under > normal operations the batch size=1 (no batching). > If a function is slow and backpressure arises, the batch size grows, though. > From an SDK perspective, this batching is not visible. The function is always > invoked with a single message. This makes it impossible to efficiently > evaluate the whole batch at once (e.g. with pandas). > This issue was requested in > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Statefun-Python-Batch-tp43022.html. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] echauchot commented on pull request #15156: [FLINK-21393] [formats] Implement ParquetAvroInputFormat
echauchot commented on pull request #15156: URL: https://github.com/apache/flink/pull/15156#issuecomment-844877632 @JingsongLi can we merge the PR? It has been 2 months since submission -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #15961: [FLINK-22706][release]Update License information in NOTICE file
zentol merged pull request #15961: URL: https://github.com/apache/flink/pull/15961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22706) The NOTICE file of Flink repo should be updated
[ https://issues.apache.org/jira/browse/FLINK-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-22706: - Priority: Major (was: Blocker) > The NOTICE file of Flink repo should be updated > --- > > Key: FLINK-22706 > URL: https://issues.apache.org/jira/browse/FLINK-22706 > Project: Flink > Issue Type: Bug > Components: Release System >Affects Versions: 1.12.1, 1.12.2, 1.13.0, 1.12.3 >Reporter: Fuyao Li >Assignee: Fuyao Li >Priority: Major > Labels: pull-request-available > Fix For: 1.13.1, 1.12.5 > > > With the PR introduced in [1], flink documentation is upgraded to Hugo > instead of Jekyll. However, we fail to update the NOTICE file. [2] > > The jQuery, bootstrap dependencies are not used anymore and AnchorJS JS > library path should also be updated. > > In addition, bootstrap version 3.3.4 is known to have security > vulnerabilities. See link [3]. This could cause legal approval rejection > while adopting Apache Flink. (I am facing such a issue.) > > I can create a pull request to fix this. Please assign the task to me. > > [1] https://issues.apache.org/jira/browse/FLINK-21193 > [2] [https://github.com/apache/flink/blob/master/NOTICE#L10] > [3]https://snyk.io/test/npm/bootstrap/3.3.4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22706) The NOTICE file of Flink repo should be updated
[ https://issues.apache.org/jira/browse/FLINK-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-22706: - Affects Version/s: (was: 1.12.3) (was: 1.12.2) (was: 1.12.1) > The NOTICE file of Flink repo should be updated > --- > > Key: FLINK-22706 > URL: https://issues.apache.org/jira/browse/FLINK-22706 > Project: Flink > Issue Type: Bug > Components: Release System >Affects Versions: 1.13.0 >Reporter: Fuyao Li >Assignee: Fuyao Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.1 > > > With the PR introduced in [1], flink documentation is upgraded to Hugo > instead of Jekyll. However, we fail to update the NOTICE file. [2] > > The jQuery, bootstrap dependencies are not used anymore and AnchorJS JS > library path should also be updated. > > In addition, bootstrap version 3.3.4 is known to have security > vulnerabilities. See link [3]. This could cause legal approval rejection > while adopting Apache Flink. (I am facing such a issue.) > > I can create a pull request to fix this. Please assign the task to me. > > [1] https://issues.apache.org/jira/browse/FLINK-21193 > [2] [https://github.com/apache/flink/blob/master/NOTICE#L10] > [3]https://snyk.io/test/npm/bootstrap/3.3.4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22706) The NOTICE file of Flink repo should be updated
[ https://issues.apache.org/jira/browse/FLINK-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-22706: - Fix Version/s: (was: 1.12.5) 1.14.0 > The NOTICE file of Flink repo should be updated > --- > > Key: FLINK-22706 > URL: https://issues.apache.org/jira/browse/FLINK-22706 > Project: Flink > Issue Type: Bug > Components: Release System >Affects Versions: 1.12.1, 1.12.2, 1.13.0, 1.12.3 >Reporter: Fuyao Li >Assignee: Fuyao Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.1 > > > With the PR introduced in [1], flink documentation is upgraded to Hugo > instead of Jekyll. However, we fail to update the NOTICE file. [2] > > The jQuery, bootstrap dependencies are not used anymore and AnchorJS JS > library path should also be updated. > > In addition, bootstrap version 3.3.4 is known to have security > vulnerabilities. See link [3]. This could cause legal approval rejection > while adopting Apache Flink. (I am facing such a issue.) > > I can create a pull request to fix this. Please assign the task to me. > > [1] https://issues.apache.org/jira/browse/FLINK-21193 > [2] [https://github.com/apache/flink/blob/master/NOTICE#L10] > [3]https://snyk.io/test/npm/bootstrap/3.3.4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-22706) The NOTICE file of Flink repo should be updated
[ https://issues.apache.org/jira/browse/FLINK-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-22706. Resolution: Fixed master: f1132f338c7bf56bff83147ab9842227c3d01e97 1.13: d1dd346aa0ce7c6300e4e392384b8ac40be0c13c > The NOTICE file of Flink repo should be updated > --- > > Key: FLINK-22706 > URL: https://issues.apache.org/jira/browse/FLINK-22706 > Project: Flink > Issue Type: Bug > Components: Release System >Affects Versions: 1.13.0 >Reporter: Fuyao Li >Assignee: Fuyao Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.1 > > > With the PR introduced in [1], flink documentation is upgraded to Hugo > instead of Jekyll. However, we fail to update the NOTICE file. [2] > > The jQuery, bootstrap dependencies are not used anymore and AnchorJS JS > library path should also be updated. > > In addition, bootstrap version 3.3.4 is known to have security > vulnerabilities. See link [3]. This could cause legal approval rejection > while adopting Apache Flink. (I am facing such a issue.) > > I can create a pull request to fix this. Please assign the task to me. > > [1] https://issues.apache.org/jira/browse/FLINK-21193 > [2] [https://github.com/apache/flink/blob/master/NOTICE#L10] > [3]https://snyk.io/test/npm/bootstrap/3.3.4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22706) Source NOTICE outdated regarding docs/
[ https://issues.apache.org/jira/browse/FLINK-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-22706: - Summary: Source NOTICE outdated regarding docs/ (was: The NOTICE file of Flink repo should be updated) > Source NOTICE outdated regarding docs/ > -- > > Key: FLINK-22706 > URL: https://issues.apache.org/jira/browse/FLINK-22706 > Project: Flink > Issue Type: Bug > Components: Release System >Affects Versions: 1.13.0 >Reporter: Fuyao Li >Assignee: Fuyao Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.1 > > > With the PR introduced in [1], flink documentation is upgraded to Hugo > instead of Jekyll. However, we fail to update the NOTICE file. [2] > > The jQuery, bootstrap dependencies are not used anymore and AnchorJS JS > library path should also be updated. > > In addition, bootstrap version 3.3.4 is known to have security > vulnerabilities. See link [3]. This could cause legal approval rejection > while adopting Apache Flink. (I am facing such a issue.) > > I can create a pull request to fix this. Please assign the task to me. > > [1] https://issues.apache.org/jira/browse/FLINK-21193 > [2] [https://github.com/apache/flink/blob/master/NOTICE#L10] > [3]https://snyk.io/test/npm/bootstrap/3.3.4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20336) RequestReplyFunction should not silently ignore UNRECOGNIZED state value mutations types
[
https://issues.apache.org/jira/browse/FLINK-20336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igal Shilman reassigned FLINK-20336:
Assignee: Tzu-Li (Gordon) Tai
> RequestReplyFunction should not silently ignore UNRECOGNIZED state value
> mutations types
>
>
> Key: FLINK-20336
> URL: https://issues.apache.org/jira/browse/FLINK-20336
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
>Affects Versions: statefun-2.1.0, statefun-2.2.1
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Labels: auto-unassigned
> Fix For: statefun-3.1.0
>
>
> If a function's response has a {{PersistedValueMutation}} type that is
> {{UNRECOGNIZED}}, we currently just silently ignore that mutation:
> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/reqreply/PersistedRemoteFunctionValues.java#L84
> This is incorrect. The {{UNRECOGNIZED}} enum constant is a pre-defined
> constant used by the Protobuf Java SDK, to represent a constant that was
> unable to be deserialized (because the the serialized constant does not match
> any enums defined in the protobuf message).
> Therefore, it should be handled by throwing an exception, preferably
> indicating that there is some sort of version mismatch between the function's
> Protobuf message definitions, and StateFun's Protobuf message definitions
> (i.e. most likely a mismatch in the invocation protocol versions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21280) Investigate a way to disable object reuse in StateFun
[ https://issues.apache.org/jira/browse/FLINK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igal Shilman updated FLINK-21280: - Fix Version/s: (was: statefun-3.1.0) > Investigate a way to disable object reuse in StateFun > - > > Key: FLINK-21280 > URL: https://issues.apache.org/jira/browse/FLINK-21280 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Priority: Major > Labels: stale-major > > Currently in StateFun, we require object reuse to be enabled. We are doing it > because Flink sources can emit arbitrary objects, but the first occurrence of > the user defined payload serializer happens in a IngressRouterOperator. This > works due to the fact that this operator is chained to the user, and with > object reuse, the source output is passed as reference (without any Flink > serialization) to the IngressRouterOperator. > Unfortunately JdbcSink requires object reuse to be disabled, and therefore > can't be used directly with StateFun. > This issue was first reported on the mailing list: > [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Stateful-Functions-JDBC-Sink-Problems-td41265.html] > > This issue is about exploring a way to remove the object reuse requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20336) RequestReplyFunction should not silently ignore UNRECOGNIZED state value mutations types
[
https://issues.apache.org/jira/browse/FLINK-20336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igal Shilman updated FLINK-20336:
-
Priority: Minor (was: Major)
> RequestReplyFunction should not silently ignore UNRECOGNIZED state value
> mutations types
>
>
> Key: FLINK-20336
> URL: https://issues.apache.org/jira/browse/FLINK-20336
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
>Affects Versions: statefun-2.1.0, statefun-2.2.1
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Minor
> Labels: auto-unassigned
> Fix For: statefun-3.1.0
>
>
> If a function's response has a {{PersistedValueMutation}} type that is
> {{UNRECOGNIZED}}, we currently just silently ignore that mutation:
> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/reqreply/PersistedRemoteFunctionValues.java#L84
> This is incorrect. The {{UNRECOGNIZED}} enum constant is a pre-defined
> constant used by the Protobuf Java SDK, to represent a constant that was
> unable to be deserialized (because the the serialized constant does not match
> any enums defined in the protobuf message).
> Therefore, it should be handled by throwing an exception, preferably
> indicating that there is some sort of version mismatch between the function's
> Protobuf message definitions, and StateFun's Protobuf message definitions
> (i.e. most likely a mismatch in the invocation protocol versions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
