[GitHub] [flink] curcur opened a new pull request #15521: [hotfix] Fix the release version for JdbcXaSinkFunction
curcur opened a new pull request #15521: URL: https://github.com/apache/flink/pull/15521 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai closed pull request #221: Update README.md
tzulitai closed pull request #221: URL: https://github.com/apache/flink-statefun/pull/221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai closed pull request #220: [hotfix][python] fix bad interpolation
tzulitai closed pull request #220: URL: https://github.com/apache/flink-statefun/pull/220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai commented on pull request #221: Update README.md
tzulitai commented on pull request #221: URL: https://github.com/apache/flink-statefun/pull/221#issuecomment-815460904 Thanks @SiddiqueAhmad. You are correct that the links are outdated due to the examples being moved to the `flink-statefun-playground` repo. I'll merge this now. As a follow-up, there is actually going to be a series of rework on the current README to reflect all the recent changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #15395: [FLINK-15146][core][ttl] Fix check that incremental cleanup size must be greater than zero
Myasuka commented on a change in pull request #15395:
URL: https://github.com/apache/flink/pull/15395#discussion_r609310327
##
File path:
flink-core/src/test/java/org/apache/flink/api/common/state/StateTtlConfigTest.java
##
@@ -71,4 +74,22 @@ public void testStateTtlConfigBuildWithCleanupInBackground()
{
assertThat(incrementalCleanupStrategy.runCleanupForEveryRecord(),
is(false));
assertThat(rocksdbCleanupStrategy.getQueryTimeAfterNumEntries(),
is(1000L));
}
+
+@Test
+public void testStateTtlConfigBuildWithNonPositiveCleanupIncrementalSize()
{
+ArrayList illegalCleanUpSizes = new ArrayList();
+illegalCleanUpSizes.add(0);
+illegalCleanUpSizes.add(-2);
Review comment:
Generally, I prefer to use `List illegalCleanUpSizes =
Arrays.asList(0, -2);` to create the candidate list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-22139) Flink Jobmanager & Task Manger logs are not writing to the logs files
[
https://issues.apache.org/jira/browse/FLINK-22139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316862#comment-17316862
]
Bhagi commented on FLINK-22139:
---
Hi Kenneth William Krugler ,
I searched in User mailing list, but did not find any solution.Hence created
this issue.
> Flink Jobmanager & Task Manger logs are not writing to the logs files
> -
>
> Key: FLINK-22139
> URL: https://issues.apache.org/jira/browse/FLINK-22139
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.2
> Environment: on kubernetes flink standalone deployment with
> jobmanager HA is enabled.
>Reporter: Bhagi
>Priority: Major
>
> Hi Team,
> I am submitting the jobs and restarting the job manager and task manager
> pods.. Log files are generating with the name task manager and job manager.
> but job manager & task manager log file size is '0', i am not sure any
> configuration missed..why logs are not writing to their log files..
> # Task Manager pod###
> flink@flink-taskmanager-85b6585b7-hhgl7:~$ ls -lart log/
> total 0
> -rw-r--r-- 1 flink flink 0 Apr 7 09:35
> flink--taskexecutor-0-flink-taskmanager-85b6585b7-hhgl7.log
> flink@flink-taskmanager-85b6585b7-hhgl7:~$
> ### Jobmanager pod Logs #
> flink@flink-jobmanager-f6db89b7f-lq4ps:~$
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:36
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-gtkx5.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:36
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-wnrfm.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:37
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-2b2fs.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:37
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-7kdhh.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 09:35
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-twhkt.log
> drwxrwxrwx 2 7148739 flink35 Apr 7 09:35 .
> -rw-r--r-- 1 7148739 flink 0 Apr 7 09:35
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-lq4ps.log
> flink@flink-jobmanager-f6db89b7f-lq4ps:~$
> I configured log4j.properties for flink
> log4j.properties: |+
> monitorInterval=30
> rootLogger.level = INFO
> rootLogger.appenderRef.file.ref = MainAppender
> logger.flink.name = org.apache.flink
> logger.flink.level = INFO
> logger.akka.name = akka
> logger.akka.level = INFO
> appender.main.name = MainAppender
> appender.main.type = RollingFile
> appender.main.append = true
> appender.main.fileName = ${sys:log.file}
> appender.main.filePattern = ${sys:log.file}.%i
> appender.main.layout.type = PatternLayout
> appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x
> - %m%n
> appender.main.policies.type = Policies
> appender.main.policies.size.type = SizeBasedTriggeringPolicy
> appender.main.policies.size.size = 100MB
> appender.main.policies.startup.type = OnStartupTriggeringPolicy
> appender.main.strategy.type = DefaultRolloverStrategy
> appender.main.strategy.max = ${env:MAX_LOG_FILE_NUMBER:-10}
> logger.netty.name =
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
> logger.netty.level = OFF
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15500: [FLINK-22077] Fix incorrect way to create cross-region ConsumedPartitionGroups in PipelinedRegionSchedulingStrategy
flinkbot edited a comment on pull request #15500: URL: https://github.com/apache/flink/pull/15500#issuecomment-814682075 ## CI report: * cc7b2e44ce50e610134a97146ecb59ea1502df39 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16109) * 444b3ecd6a28d7dc31c079f9def61846692c8528 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16177) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
flinkbot edited a comment on pull request #15307: URL: https://github.com/apache/flink/pull/15307#issuecomment-803592550 ## CI report: * 2ef40343abecf693b39b7b7fcb9f6d0d26bf82cd UNKNOWN * f7dde849a55b4aad1044ac200b390a604029c710 UNKNOWN * a3341911ab289f00a4aa50b99c8b90980f5fe453 UNKNOWN * d0b9eaf2a807fcec1d0bb54673ba0572390b4c31 UNKNOWN * 24d902609498c22d246a2d8618da856ae3b39a2e UNKNOWN * 3853944d77be88a1657f80208e82b9108e944b46 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16168) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15500: [FLINK-22077] Fix incorrect way to create cross-region ConsumedPartitionGroups in PipelinedRegionSchedulingStrategy
flinkbot edited a comment on pull request #15500: URL: https://github.com/apache/flink/pull/15500#issuecomment-814682075 ## CI report: * cc7b2e44ce50e610134a97146ecb59ea1502df39 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16109) * 444b3ecd6a28d7dc31c079f9def61846692c8528 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15121: The method $(String) is undefined for the type TableExample
flinkbot edited a comment on pull request #15121: URL: https://github.com/apache/flink/pull/15121#issuecomment-793480240 ## CI report: * cf26ce895a7956d258acc073817f578558e78227 UNKNOWN * 709c4110370b845f42d916a06e56c6026cf2fac8 UNKNOWN * 4ea99332e1997eebca1f3f0a9d9229b8265fe32c UNKNOWN * 69cd36d9aa6e4aeb2ad827020d125712307ab585 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16167) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17957) Forbidden syntax "CREATE SYSTEM FUNCTION" for sql parser
[ https://issues.apache.org/jira/browse/FLINK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-17957. --- Resolution: Fixed Fixed in master: 43feb550f080c4d927cebc066a68f6d177800f26 > Forbidden syntax "CREATE SYSTEM FUNCTION" for sql parser > > > Key: FLINK-17957 > URL: https://issues.apache.org/jira/browse/FLINK-17957 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: Danny Chen >Assignee: WeiNan Zhao >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > This syntax is invalid, but the parser still works. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #15494: [FLINK-17957] [table-api][parser] Forbidden syntax "CREATE SYSTEM FUNCTION" for sql parser
wuchong merged pull request #15494: URL: https://github.com/apache/flink/pull/15494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #15494: [FLINK-17957] [table-api][parser] Forbidden syntax "CREATE SYSTEM FUNCTION" for sql parser
wuchong commented on pull request #15494: URL: https://github.com/apache/flink/pull/15494#issuecomment-815439178 @chaozwn , thanks for the contribution. However, please note to fill in the PR description, and please do not squash commits, otherwise, it's hard to know which part is changed since last review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21643) JDBC sink should be able to execute statements on multiple tables
[ https://issues.apache.org/jira/browse/FLINK-21643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-21643: Affects Version/s: (was: 1.13.0) > JDBC sink should be able to execute statements on multiple tables > - > > Key: FLINK-21643 > URL: https://issues.apache.org/jira/browse/FLINK-21643 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Maciej Obuchowski >Priority: Major > Labels: pull-request-available > > Currently datastream JDBC sink supports outputting data only to one table - > by having to provide SQL template, from which SimpleBatchStatementExecutor > creates PreparedStatement. Creating multiple sinks, each of which writes data > to one table is impractical for moderate to large number of tables - > relational databases don't usually tolerate large number of connections. > I propose adding DynamicBatchStatementExecutor, which will additionally > require > 1) provided mechanism to create SQL statements based on given object > 2) cache for prepared statements > 3) mechanism for determining which statement should be used for given object -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Thesharing commented on pull request #15500: [FLINK-22077] Fix incorrect way to create cross-region ConsumedPartitionGroups in PipelinedRegionSchedulingStrategy
Thesharing commented on pull request #15500: URL: https://github.com/apache/flink/pull/15500#issuecomment-815435726 > Thanks for creating this PR @Thesharing. I think the change looks good to me. > > Before merging this PR we should add a test which ensures the correct computation of the cross-region `ConsumedPartitionGroups`. This will help us to not break this functionality. Would such a test be possible to add @Thesharing? Thank you so much, @tillrohrmann. I've added two test cases for the computation of cross-region `ConsumedPartitionGroup`s. The first one makes sure that cross-region `ConsumedPartitionGroup`s are calculated correctly in the graph illustrated in FLINK-22017. The second one makes sure that there's no cross-region `ConsumedPartitionGroup` for all-to-all blocking edges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] SiddiqueAhmad opened a new pull request #221: Update README.md
SiddiqueAhmad opened a new pull request #221: URL: https://github.com/apache/flink-statefun/pull/221 links are not updated, i feel they are moved in playground repo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15501: [FLINK-22054][k8s] Using a shared watcher for ConfigMap watching
flinkbot edited a comment on pull request #15501: URL: https://github.com/apache/flink/pull/15501#issuecomment-814722337 ## CI report: * 908d66e15d486c52662d9757950f78ec682506e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16176) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16140) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19363) Code of split method grows beyond 64 KB
[
https://issues.apache.org/jira/browse/FLINK-19363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated FLINK-19363:
---
Component/s: (was: Table SQL / Planner)
Table SQL / Runtime
> Code of split method grows beyond 64 KB
> ---
>
> Key: FLINK-19363
> URL: https://issues.apache.org/jira/browse/FLINK-19363
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: hailong wang
>Priority: Major
> Fix For: 1.14.0
>
>
> For now, If the total size beyond 64kb, we will split one field into one
> field. But when sql is complicated, one field size will grow 64kb.
> {code:java}
> Caused by: org.codehaus.janino.InternalCompilerException: Compiling
> "StreamExecCalc$4436": Code of method "split$4435$(LStreamExecCalc$4436;)V"
> of class "StreamExecCalc$4436" grows beyond 64 KBCaused by:
> org.codehaus.janino.InternalCompilerException: Compiling
> "StreamExecCalc$4436": Code of method "split$4435$(LStreamExecCalc$4436;)V"
> of class "StreamExecCalc$4436" grows beyond 64 KB at
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382) at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237) at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216) at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207) at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80) at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75) at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:78)
> ... 24 moreCaused by: org.codehaus.janino.InternalCompilerException: Code of
> method "split$4435$(LStreamExecCalc$4436;)V" of class "StreamExecCalc$4436"
> grows beyond 64 KB at
> org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:1009) at
> org.codehaus.janino.CodeContext.write(CodeContext.java:918) at
> org.codehaus.janino.CodeContext.writeBranch(CodeContext.java:1038)
> {code}
> By this case, we should split one field to multi functions.
> My solution is as follow,
> {code:java}
> override def visitCall(call: RexCall): GeneratedExpression = {
> val resultType = FlinkTypeFactory.toLogicalType(call.getType)
> // convert operands and help giving untyped NULL literals a type
> val operands = call.getOperands.zipWithIndex.map {
> // this helps e.g. for AS(null)
> // we might need to extend this logic in case some rules do not create
> typed NULLs
> case (operandLiteral: RexLiteral, 0) if
> operandLiteral.getType.getSqlTypeName == SqlTypeName.NULL &&
> call.getOperator.getReturnTypeInference == ReturnTypes.ARG0 =>
> generateNullLiteral(resultType, ctx.nullCheck)
> case (o@_, _) => o.accept(this)
> }
> // when function is too large, we split it.
> val exp = generateCallExpression(ctx, call.getOperator, operands,
> resultType)
> if (exp.code.length > maxGeneratedCodeLength) {
> ctx.setCodeSplit()
> val methodName = newName("callSplit")
> val resultTypeTerm = boxedTypeTermForType(exp.resultType)
> val callResultTerm = ctx.addReusableLocalVariable(resultTypeTerm,
> "callSplitResultTerm")
> val callNullResultTerm = ctx.addReusableLocalVariable("boolean",
> "callSplitNullResultTerm")
> val method =
> s"""
> |private void $methodName() throws Exception {
> | ${exp.code}
> | ${callResultTerm}= ${exp.resultTerm};
> | ${callNullResultTerm}= ${exp.nullTerm};
> |}
> |""".stripMargin
> val methodCode =
> s"""
> |$methodName();
> |""".stripMargin
> ctx.addReusableMember(method)
> return new GeneratedExpression(callResultTerm, callNullResultTerm,
> methodCode, exp.resultType)
> }
> exp
> }
> {code}
> When function beyong maxGeneratedCodeLength, we split it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19066) InnerJoinITCase.testBigForSpill failed with "java.lang.ClassCastException: org.apache.flink.table.runtime.util.ResettableExternalBuffer$BufferIterator cannot be cast to
[
https://issues.apache.org/jira/browse/FLINK-19066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated FLINK-19066:
---
Component/s: (was: Table SQL / Planner)
Table SQL / Runtime
> InnerJoinITCase.testBigForSpill failed with "java.lang.ClassCastException:
> org.apache.flink.table.runtime.util.ResettableExternalBuffer$BufferIterator
> cannot be cast to org.apache.flink.table.data.binary.BinaryRowData"
> ---
>
> Key: FLINK-19066
> URL: https://issues.apache.org/jira/browse/FLINK-19066
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Caizhi Weng
>Priority: Major
> Labels: test-stability
> Fix For: 1.13.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5942=logs=904e5037-64c0-5f69-f6d5-e21b89cf6484=39857031-7f0c-5fd5-d730-a19c5794f839]
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.flink.table.runtime.util.ResettableExternalBuffer$BufferIterator
> cannot be cast to org.apache.flink.table.data.binary.BinaryRowData
> at
> org.apache.flink.table.runtime.util.ResettableExternalBuffer$InMemoryBuffer$InMemoryBufferIterator.next(ResettableExternalBuffer.java:678)
> at
> org.apache.flink.table.runtime.util.ResettableExternalBuffer$InMemoryBuffer$InMemoryBufferIterator.next(ResettableExternalBuffer.java:650)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] curcur closed pull request #429: [FLINK-22004] Translate Flink Roadmap to Chinese
curcur closed pull request #429: URL: https://github.com/apache/flink-web/pull/429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] curcur commented on pull request #429: [FLINK-22004] Translate Flink Roadmap to Chinese
curcur commented on pull request #429: URL: https://github.com/apache/flink-web/pull/429#issuecomment-815429333 should pull against apache:asf-site instead of apache:master, close this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 edited a comment on pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
lindong28 edited a comment on pull request #15518: URL: https://github.com/apache/flink/pull/15518#issuecomment-815422325 For the context, @tillrohrmann asked in this comment[1] about the excessive INFO level logging in AZP. It appears that INFO level logging from org.apache.flink.streaming.connectors.kafka are printed in AZP due to the following reasons: - By searching for "log4j.configurationFile" in [2], we could see that `tools/ci/log4j.properties` is used to configure the log4j level in AZP. - `org.apache.flink.streaming.connectors.kafka` is specifically updated to use ConsoleAppender instead of FileAppender inside the file `tools/ci/log4j.properties` I guess the questions are: - Is it expected that AZP test uses `tools/ci/log4j.properties` instead of `src/test/log4j2-test.properties` for log4j configuration? - Why do we specifically set `org.apache.flink.streaming.connectors.kafka` to use `ConsoleAppender` in `tools/ci/log4j.properties`? @rmetzger, @zentol: do you have any suggestion? Thanks. [1] https://issues.apache.org/jira/browse/FLINK-18634?focusedCommentId=17316208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17316208 [2] https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16080=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] curcur opened a new pull request #430: [FLINK-22004] Update Flink Roadmap Chinese Version
curcur opened a new pull request #430: URL: https://github.com/apache/flink-web/pull/430 1. Translate Flink Roadmap to Chinese 2. Feature Radar (Figure) may also need to be translated/updated 3. Already built locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22147) Refactor Partition Discovery Logic in KafkaSourceEnumerator
Qingsheng Ren created FLINK-22147:
-
Summary: Refactor Partition Discovery Logic in
KafkaSourceEnumerator
Key: FLINK-22147
URL: https://issues.apache.org/jira/browse/FLINK-22147
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Qingsheng Ren
Currently the logic of partition discovery is: the worker thread checks if
there's new partitions and initialize new splits if so, then coordinator thread
marks these splits as pending and try to make assignments.
Under current design, the worker thread needs to keep an internal data
structure tracking already discovered partitions, which is duplicated with
pending splits + assigned partitions tracked by coordinator thread. Usually
this kind of double-bookkeeping is fragile.
Another issue is that the worker thread always fetches descriptions of ALL
topics at partition discovery, which will comes to a problem working with a
giant Kafka clusters with millions of topics/partitions.
In order to fix issues above, a refactor is needed for the partition discovery
logic in Kafka enumerator. Basically the logic can be changed to:
# The worker thread fetches descriptions of subscribed topics/partitions, then
hands over to coordinator thread
# The coordinator thread filters out already discovered partitions (pending +
assigned partitions), then invokes worker thread with {{callAsync}} to fetch
offsets for new partitions
# The worker thread fetches offsets and creates splits for new partitions,
then hands over new splits to coordinator thread
# The coordinator thread marks these splits as pending and try to make
assignment.
Discussion of this issue can be found in
[https://github.com/apache/flink/pull/15461] .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] fsk119 commented on a change in pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
fsk119 commented on a change in pull request #15307:
URL: https://github.com/apache/flink/pull/15307#discussion_r609250053
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushFilterInCalcIntoTableSourceScanRule.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.logical;
+
+import
org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown;
+import
org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.FlinkPreparingTableBase;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.core.Calc;
+import org.apache.calcite.rel.logical.LogicalCalc;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalTableScan;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexProgramBuilder;
+import org.apache.calcite.tools.RelBuilder;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+/**
+ * Pushes a {@link LogicalFilter} from the {@link LogicalCalc} and into a
{@link LogicalTableScan}.
+ */
+public class PushFilterInCalcIntoTableSourceScanRule extends
PushFilterIntoSourceScanRuleBase {
+public static final PushFilterInCalcIntoTableSourceScanRule INSTANCE =
+new PushFilterInCalcIntoTableSourceScanRule();
+
+public PushFilterInCalcIntoTableSourceScanRule() {
+super(
+operand(Calc.class, operand(FlinkLogicalTableSourceScan.class,
none())),
+"PushFilterInCalcIntoTableSourceScanRule");
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+super.matches(call);
+
+Calc calc = call.rel(0);
+RexProgram originProgram = calc.getProgram();
+
+if (originProgram.getCondition() == null) {
+return false;
+}
+
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable tableSourceTable =
scan.getTable().unwrap(TableSourceTable.class);
+// we can not push filter twice
+return canPushdownFilter(tableSourceTable);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+Calc calc = call.rel(0);
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable table =
scan.getTable().unwrap(TableSourceTable.class);
+pushFilterIntoScan(call, calc, scan, table);
+}
+
+private void pushFilterIntoScan(
+RelOptRuleCall call,
+Calc calc,
+FlinkLogicalTableSourceScan scan,
+FlinkPreparingTableBase relOptTable) {
+
+RexProgram originProgram = calc.getProgram();
+
+RelBuilder relBuilder = call.builder();
+Tuple2 extractedPredicates =
+extractPredicates(
+
originProgram.getInputRowType().getFieldNames().toArray(new String[0]),
+
originProgram.expandLocalRef(originProgram.getCondition()),
+scan,
+relBuilder.getRexBuilder());
+
+RexNode[] convertiblePredicates = extractedPredicates._1;
+if (convertiblePredicates.length == 0) {
+// no condition can be translated to expression
+return;
+}
+
+Tuple2
pushdownResultWithScan =
+createTableScanAfterPushdown(
+convertiblePredicates,
+relOptTable.unwrap(TableSourceTable.class),
+scan,
+relBuilder);
+
+SupportsFilterPushDown.Result result = pushdownResultWithScan._1;
+TableSourceTable tableSourceTable = pushdownResultWithScan._2;
+
+FlinkLogicalTableSourceScan newScan =
+FlinkLogicalTableSourceScan.create(scan.getCluster(),
tableSourceTable);
+
+// build new calc program
+RexProgramBuilder programBuilder =
+new
[GitHub] [flink] flinkbot edited a comment on pull request #15520: [FLINK-22097][sql-client] CliResultView should wait RefreshThread exi…
flinkbot edited a comment on pull request #15520: URL: https://github.com/apache/flink/pull/15520#issuecomment-815418058 ## CI report: * 41dd8818b80689f45e6b24f5e7ad4edbe7b7ecf7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16175) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15501: [FLINK-22054][k8s] Using a shared watcher for ConfigMap watching
flinkbot edited a comment on pull request #15501: URL: https://github.com/apache/flink/pull/15501#issuecomment-814722337 ## CI report: * 908d66e15d486c52662d9757950f78ec682506e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16140) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16176) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a change in pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
fsk119 commented on a change in pull request #15307:
URL: https://github.com/apache/flink/pull/15307#discussion_r608457950
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushFilterInCalcIntoTableSourceScanRule.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.logical;
+
+import
org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown;
+import
org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.FlinkPreparingTableBase;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.core.Calc;
+import org.apache.calcite.rel.logical.LogicalCalc;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalTableScan;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexProgramBuilder;
+import org.apache.calcite.tools.RelBuilder;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+/**
+ * Pushes a {@link LogicalFilter} from the {@link LogicalCalc} and into a
{@link LogicalTableScan}.
+ */
+public class PushFilterInCalcIntoTableSourceScanRule extends
PushFilterIntoSourceScanRuleBase {
+public static final PushFilterInCalcIntoTableSourceScanRule INSTANCE =
+new PushFilterInCalcIntoTableSourceScanRule();
+
+public PushFilterInCalcIntoTableSourceScanRule() {
+super(
+operand(Calc.class, operand(FlinkLogicalTableSourceScan.class,
none())),
+"PushFilterInCalcIntoTableSourceScanRule");
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+super.matches(call);
+
+Calc calc = call.rel(0);
+RexProgram originProgram = calc.getProgram();
+
+if (originProgram.getCondition() == null) {
+return false;
+}
+
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable tableSourceTable =
scan.getTable().unwrap(TableSourceTable.class);
+// we can not push filter twice
+return canPushdownFilter(tableSourceTable);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+Calc calc = call.rel(0);
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable table =
scan.getTable().unwrap(TableSourceTable.class);
+pushFilterIntoScan(call, calc, scan, table);
+}
+
+private void pushFilterIntoScan(
+RelOptRuleCall call,
+Calc calc,
+FlinkLogicalTableSourceScan scan,
+FlinkPreparingTableBase relOptTable) {
+
+RexProgram originProgram = calc.getProgram();
+
+RelBuilder relBuilder = call.builder();
+Tuple2 extractedPredicates =
+extractPredicates(
+
originProgram.getInputRowType().getFieldNames().toArray(new String[0]),
+
originProgram.expandLocalRef(originProgram.getCondition()),
+scan,
+relBuilder.getRexBuilder());
+
+RexNode[] convertiblePredicates = extractedPredicates._1;
+if (convertiblePredicates.length == 0) {
+// no condition can be translated to expression
+return;
+}
+
+Tuple2
pushdownResultWithScan =
+createTableScanAfterPushdown(
+convertiblePredicates,
+relOptTable.unwrap(TableSourceTable.class),
+scan,
+relBuilder);
+
+SupportsFilterPushDown.Result result = pushdownResultWithScan._1;
+TableSourceTable tableSourceTable = pushdownResultWithScan._2;
+
+FlinkLogicalTableSourceScan newScan =
+FlinkLogicalTableSourceScan.create(scan.getCluster(),
tableSourceTable);
+
+// build new calc program
+RexProgramBuilder programBuilder =
+new
[GitHub] [flink] yittg commented on pull request #15501: [FLINK-22054][k8s] Using a shared watcher for ConfigMap watching
yittg commented on pull request #15501: URL: https://github.com/apache/flink/pull/15501#issuecomment-815422875 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 commented on pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
lindong28 commented on pull request #15518: URL: https://github.com/apache/flink/pull/15518#issuecomment-815422325 For the context, @tillrohrmann asked in this comment about the excessive INFO level logging in AZP. It appears that INFO level logging from org.apache.flink.streaming.connectors.kafka are printed in AZP due to the following reasons: - By searching for "log4j.configurationFile", we could see that `tools/ci/log4j.properties` is used to configure the log4j level in AZP. - `org.apache.flink.streaming.connectors.kafka` is specifically updated to use ConsoleAppender instead of FileAppender inside the file `tools/ci/log4j.properties` I guess the questions are: - Is it expected that AZP test uses `tools/ci/log4j.properties` instead of `src/test/log4j2-test.properties` for log4j configuration? - Why do we specifically set `org.apache.flink.streaming.connectors.kafka` to use `ConsoleAppender` in `tools/ci/log4j.properties`? @rmetzger, @zentol: do you have any suggestion? Thanks. [1] https://issues.apache.org/jira/browse/FLINK-18634?focusedCommentId=17316208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17316208 [2] https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16080=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 removed a comment on pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
lindong28 removed a comment on pull request #15518: URL: https://github.com/apache/flink/pull/15518#issuecomment-815419610 @rmetzger -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 commented on pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
lindong28 commented on pull request #15518: URL: https://github.com/apache/flink/pull/15518#issuecomment-815419610 @rmetzger -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-20663) Managed memory may not be released in time when operators use managed memory frequently
[
https://issues.apache.org/jira/browse/FLINK-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316842#comment-17316842
]
zhou edited comment on FLINK-20663 at 4/8/21, 3:27 AM:
---
logs in attachment,it a long text: [^exception]
was (Author: zhou_yb):
logs in attachment: [^exception]
> Managed memory may not be released in time when operators use managed memory
> frequently
> ---
>
> Key: FLINK-20663
> URL: https://issues.apache.org/jira/browse/FLINK-20663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Caizhi Weng
>Assignee: Xintong Song
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.2, 1.13.0
>
> Attachments: exception
>
>
> Some batch operators (like sort merge join or hash aggregate) use managed
> memory frequently. When these operators are chained together and the cluster
> load is a bit heavy, it is very likely that the following exception occurs:
> {code:java}
> 2020-12-18 10:04:32
> java.lang.RuntimeException:
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate
> 512 pages
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:85)
> at
> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.(SimpleCollectingOutputView.java:49)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.(BytesHashMap.java:297)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:103)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:90)
> at LocalHashAggregateWithKeys$209161.open(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:506)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:834)
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209161.close(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.TableStreamOperator.dispose(TableStreamOperator.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:739)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:719)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:642)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:551)
> ... 3 more
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209766.close(Unknown
> Source)
> ... 8 more
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 512 pages
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:231)
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:83)
> ... 13 more
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 16777216 bytes, only 9961487 bytes are remaining. This usually
> indicates that you are requesting more memory than you have reserved.
> However, when running an old JVM version it can also be caused by slow
> garbage collection. Try to upgrade to Java 8u72 or higher if running on an
> old Java version.
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:164)
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:80)
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:229)
> ... 14 more
> {code}
> It seems that this is caused by relying on GC to release managed memory, as
> {{System.gc()}} may not trigger GC in time. See {{UnsafeMemoryBudget.java}}.
--
This message was sent by Atlassian Jira
[GitHub] [flink] flinkbot commented on pull request #15520: [FLINK-22097][sql-client] CliResultView should wait RefreshThread exi…
flinkbot commented on pull request #15520: URL: https://github.com/apache/flink/pull/15520#issuecomment-815418058 ## CI report: * 41dd8818b80689f45e6b24f5e7ad4edbe7b7ecf7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15519: [FLINK-22120] Remove duplicate code in generated code for map get
flinkbot edited a comment on pull request #15519: URL: https://github.com/apache/flink/pull/15519#issuecomment-815409179 ## CI report: * 41890dd12eec1ca6aa45867999a5fb6d1895cd9b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16174) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a change in pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
fsk119 commented on a change in pull request #15307:
URL: https://github.com/apache/flink/pull/15307#discussion_r609250053
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushFilterInCalcIntoTableSourceScanRule.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.logical;
+
+import
org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown;
+import
org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.FlinkPreparingTableBase;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.core.Calc;
+import org.apache.calcite.rel.logical.LogicalCalc;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalTableScan;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexProgramBuilder;
+import org.apache.calcite.tools.RelBuilder;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+/**
+ * Pushes a {@link LogicalFilter} from the {@link LogicalCalc} and into a
{@link LogicalTableScan}.
+ */
+public class PushFilterInCalcIntoTableSourceScanRule extends
PushFilterIntoSourceScanRuleBase {
+public static final PushFilterInCalcIntoTableSourceScanRule INSTANCE =
+new PushFilterInCalcIntoTableSourceScanRule();
+
+public PushFilterInCalcIntoTableSourceScanRule() {
+super(
+operand(Calc.class, operand(FlinkLogicalTableSourceScan.class,
none())),
+"PushFilterInCalcIntoTableSourceScanRule");
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+super.matches(call);
+
+Calc calc = call.rel(0);
+RexProgram originProgram = calc.getProgram();
+
+if (originProgram.getCondition() == null) {
+return false;
+}
+
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable tableSourceTable =
scan.getTable().unwrap(TableSourceTable.class);
+// we can not push filter twice
+return canPushdownFilter(tableSourceTable);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+Calc calc = call.rel(0);
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable table =
scan.getTable().unwrap(TableSourceTable.class);
+pushFilterIntoScan(call, calc, scan, table);
+}
+
+private void pushFilterIntoScan(
+RelOptRuleCall call,
+Calc calc,
+FlinkLogicalTableSourceScan scan,
+FlinkPreparingTableBase relOptTable) {
+
+RexProgram originProgram = calc.getProgram();
+
+RelBuilder relBuilder = call.builder();
+Tuple2 extractedPredicates =
+extractPredicates(
+
originProgram.getInputRowType().getFieldNames().toArray(new String[0]),
+
originProgram.expandLocalRef(originProgram.getCondition()),
+scan,
+relBuilder.getRexBuilder());
+
+RexNode[] convertiblePredicates = extractedPredicates._1;
+if (convertiblePredicates.length == 0) {
+// no condition can be translated to expression
+return;
+}
+
+Tuple2
pushdownResultWithScan =
+createTableScanAfterPushdown(
+convertiblePredicates,
+relOptTable.unwrap(TableSourceTable.class),
+scan,
+relBuilder);
+
+SupportsFilterPushDown.Result result = pushdownResultWithScan._1;
+TableSourceTable tableSourceTable = pushdownResultWithScan._2;
+
+FlinkLogicalTableSourceScan newScan =
+FlinkLogicalTableSourceScan.create(scan.getCluster(),
tableSourceTable);
+
+// build new calc program
+RexProgramBuilder programBuilder =
+new
[GitHub] [flink] flinkbot edited a comment on pull request #15498: [FLINK-22121][table-planner-blink] FlinkLogicalRankRuleBase now check if name of rankNumberType already exists in the input
flinkbot edited a comment on pull request #15498: URL: https://github.com/apache/flink/pull/15498#issuecomment-814575824 ## CI report: * c6fc2d4fa4d1149f364432891223779d23a8f6a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16103) * ac009d9c49b7442cedee7050b503cf9164ee2683 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16173) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20663) Managed memory may not be released in time when operators use managed memory frequently
[
https://issues.apache.org/jira/browse/FLINK-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316842#comment-17316842
]
zhou commented on FLINK-20663:
--
logs in attachment: [^exception]
> Managed memory may not be released in time when operators use managed memory
> frequently
> ---
>
> Key: FLINK-20663
> URL: https://issues.apache.org/jira/browse/FLINK-20663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Caizhi Weng
>Assignee: Xintong Song
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.2, 1.13.0
>
> Attachments: exception
>
>
> Some batch operators (like sort merge join or hash aggregate) use managed
> memory frequently. When these operators are chained together and the cluster
> load is a bit heavy, it is very likely that the following exception occurs:
> {code:java}
> 2020-12-18 10:04:32
> java.lang.RuntimeException:
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate
> 512 pages
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:85)
> at
> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.(SimpleCollectingOutputView.java:49)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.(BytesHashMap.java:297)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:103)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:90)
> at LocalHashAggregateWithKeys$209161.open(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:506)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:834)
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209161.close(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.TableStreamOperator.dispose(TableStreamOperator.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:739)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:719)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:642)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:551)
> ... 3 more
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209766.close(Unknown
> Source)
> ... 8 more
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 512 pages
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:231)
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:83)
> ... 13 more
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 16777216 bytes, only 9961487 bytes are remaining. This usually
> indicates that you are requesting more memory than you have reserved.
> However, when running an old JVM version it can also be caused by slow
> garbage collection. Try to upgrade to Java 8u72 or higher if running on an
> old Java version.
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:164)
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:80)
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:229)
> ... 14 more
> {code}
> It seems that this is caused by relying on GC to release managed memory, as
> {{System.gc()}} may not trigger GC in time. See {{UnsafeMemoryBudget.java}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20663) Managed memory may not be released in time when operators use managed memory frequently
[
https://issues.apache.org/jira/browse/FLINK-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou updated FLINK-20663:
-
Attachment: exception
> Managed memory may not be released in time when operators use managed memory
> frequently
> ---
>
> Key: FLINK-20663
> URL: https://issues.apache.org/jira/browse/FLINK-20663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Caizhi Weng
>Assignee: Xintong Song
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.2, 1.13.0
>
> Attachments: exception
>
>
> Some batch operators (like sort merge join or hash aggregate) use managed
> memory frequently. When these operators are chained together and the cluster
> load is a bit heavy, it is very likely that the following exception occurs:
> {code:java}
> 2020-12-18 10:04:32
> java.lang.RuntimeException:
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate
> 512 pages
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:85)
> at
> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.(SimpleCollectingOutputView.java:49)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.(BytesHashMap.java:297)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:103)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:90)
> at LocalHashAggregateWithKeys$209161.open(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:506)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:834)
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209161.close(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.TableStreamOperator.dispose(TableStreamOperator.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:739)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:719)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:642)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:551)
> ... 3 more
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209766.close(Unknown
> Source)
> ... 8 more
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 512 pages
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:231)
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:83)
> ... 13 more
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 16777216 bytes, only 9961487 bytes are remaining. This usually
> indicates that you are requesting more memory than you have reserved.
> However, when running an old JVM version it can also be caused by slow
> garbage collection. Try to upgrade to Java 8u72 or higher if running on an
> old Java version.
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:164)
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:80)
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:229)
> ... 14 more
> {code}
> It seems that this is caused by relying on GC to release managed memory, as
> {{System.gc()}} may not trigger GC in time. See {{UnsafeMemoryBudget.java}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
flinkbot edited a comment on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-814453955 ## CI report: * f39c534bfa814d33dbe0646fe79c09f5b5cc32a9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16164) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15489: [FLINK-22106][table-planner-blink] Result type of GeneratedExpression in StringCallGen should be compatible with their definition in
flinkbot edited a comment on pull request #15489: URL: https://github.com/apache/flink/pull/15489#issuecomment-813835360 ## CI report: * 0bd5b6f241ae6168c0abb82873f4d47448e1330c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16110) * 71bb2d2f86ca53b19dbf181ecdaa89ab4e83d44e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16172) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a change in pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
fsk119 commented on a change in pull request #15307:
URL: https://github.com/apache/flink/pull/15307#discussion_r609250053
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushFilterInCalcIntoTableSourceScanRule.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.logical;
+
+import
org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown;
+import
org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.FlinkPreparingTableBase;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.core.Calc;
+import org.apache.calcite.rel.logical.LogicalCalc;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalTableScan;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexProgramBuilder;
+import org.apache.calcite.tools.RelBuilder;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+/**
+ * Pushes a {@link LogicalFilter} from the {@link LogicalCalc} and into a
{@link LogicalTableScan}.
+ */
+public class PushFilterInCalcIntoTableSourceScanRule extends
PushFilterIntoSourceScanRuleBase {
+public static final PushFilterInCalcIntoTableSourceScanRule INSTANCE =
+new PushFilterInCalcIntoTableSourceScanRule();
+
+public PushFilterInCalcIntoTableSourceScanRule() {
+super(
+operand(Calc.class, operand(FlinkLogicalTableSourceScan.class,
none())),
+"PushFilterInCalcIntoTableSourceScanRule");
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+super.matches(call);
+
+Calc calc = call.rel(0);
+RexProgram originProgram = calc.getProgram();
+
+if (originProgram.getCondition() == null) {
+return false;
+}
+
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable tableSourceTable =
scan.getTable().unwrap(TableSourceTable.class);
+// we can not push filter twice
+return canPushdownFilter(tableSourceTable);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+Calc calc = call.rel(0);
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable table =
scan.getTable().unwrap(TableSourceTable.class);
+pushFilterIntoScan(call, calc, scan, table);
+}
+
+private void pushFilterIntoScan(
+RelOptRuleCall call,
+Calc calc,
+FlinkLogicalTableSourceScan scan,
+FlinkPreparingTableBase relOptTable) {
+
+RexProgram originProgram = calc.getProgram();
+
+RelBuilder relBuilder = call.builder();
+Tuple2 extractedPredicates =
+extractPredicates(
+
originProgram.getInputRowType().getFieldNames().toArray(new String[0]),
+
originProgram.expandLocalRef(originProgram.getCondition()),
+scan,
+relBuilder.getRexBuilder());
+
+RexNode[] convertiblePredicates = extractedPredicates._1;
+if (convertiblePredicates.length == 0) {
+// no condition can be translated to expression
+return;
+}
+
+Tuple2
pushdownResultWithScan =
+createTableScanAfterPushdown(
+convertiblePredicates,
+relOptTable.unwrap(TableSourceTable.class),
+scan,
+relBuilder);
+
+SupportsFilterPushDown.Result result = pushdownResultWithScan._1;
+TableSourceTable tableSourceTable = pushdownResultWithScan._2;
+
+FlinkLogicalTableSourceScan newScan =
+FlinkLogicalTableSourceScan.create(scan.getCluster(),
tableSourceTable);
+
+// build new calc program
+RexProgramBuilder programBuilder =
+new
[GitHub] [flink] flinkbot edited a comment on pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
flinkbot edited a comment on pull request #15307: URL: https://github.com/apache/flink/pull/15307#issuecomment-803592550 ## CI report: * 2ef40343abecf693b39b7b7fcb9f6d0d26bf82cd UNKNOWN * f7dde849a55b4aad1044ac200b390a604029c710 UNKNOWN * a3341911ab289f00a4aa50b99c8b90980f5fe453 UNKNOWN * d0b9eaf2a807fcec1d0bb54673ba0572390b4c31 UNKNOWN * 24d902609498c22d246a2d8618da856ae3b39a2e UNKNOWN * 23bca0d519a07b94b82676ac2c619271f4674051 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16163) * 3853944d77be88a1657f80208e82b9108e944b46 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16168) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a change in pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
fsk119 commented on a change in pull request #15307:
URL: https://github.com/apache/flink/pull/15307#discussion_r609247479
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushFilterInCalcIntoTableSourceScanRule.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.logical;
+
+import
org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown;
+import
org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.FlinkPreparingTableBase;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.core.Calc;
+import org.apache.calcite.rel.logical.LogicalCalc;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalTableScan;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexProgramBuilder;
+import org.apache.calcite.tools.RelBuilder;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+/**
+ * Pushes a {@link LogicalFilter} from the {@link LogicalCalc} and into a
{@link LogicalTableScan}.
+ */
+public class PushFilterInCalcIntoTableSourceScanRule extends
PushFilterIntoSourceScanRuleBase {
+public static final PushFilterInCalcIntoTableSourceScanRule INSTANCE =
+new PushFilterInCalcIntoTableSourceScanRule();
+
+public PushFilterInCalcIntoTableSourceScanRule() {
+super(
+operand(Calc.class, operand(FlinkLogicalTableSourceScan.class,
none())),
+"PushFilterInCalcIntoTableSourceScanRule");
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+super.matches(call);
+
+Calc calc = call.rel(0);
+RexProgram originProgram = calc.getProgram();
+
+if (originProgram.getCondition() == null) {
+return false;
+}
+
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable tableSourceTable =
scan.getTable().unwrap(TableSourceTable.class);
+// we can not push filter twice
+return canPushdownFilter(tableSourceTable);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+Calc calc = call.rel(0);
+FlinkLogicalTableSourceScan scan = call.rel(1);
+TableSourceTable table =
scan.getTable().unwrap(TableSourceTable.class);
+pushFilterIntoScan(call, calc, scan, table);
+}
+
+private void pushFilterIntoScan(
+RelOptRuleCall call,
+Calc calc,
+FlinkLogicalTableSourceScan scan,
+FlinkPreparingTableBase relOptTable) {
+
+RexProgram originProgram = calc.getProgram();
+
+RelBuilder relBuilder = call.builder();
+Tuple2 extractedPredicates =
+extractPredicates(
+
originProgram.getInputRowType().getFieldNames().toArray(new String[0]),
+
originProgram.expandLocalRef(originProgram.getCondition()),
+scan,
+relBuilder.getRexBuilder());
+
+RexNode[] convertiblePredicates = extractedPredicates._1;
+if (convertiblePredicates.length == 0) {
+// no condition can be translated to expression
+return;
+}
+
+Tuple2
pushdownResultWithScan =
+createTableScanAfterPushdown(
+convertiblePredicates,
+relOptTable.unwrap(TableSourceTable.class),
+scan,
+relBuilder);
+
+SupportsFilterPushDown.Result result = pushdownResultWithScan._1;
+TableSourceTable tableSourceTable = pushdownResultWithScan._2;
+
+FlinkLogicalTableSourceScan newScan =
+FlinkLogicalTableSourceScan.create(scan.getCluster(),
tableSourceTable);
+
+// build new calc program
+RexProgramBuilder programBuilder =
+new
[GitHub] [flink] flinkbot commented on pull request #15520: [FLINK-22097][sql-client] CliResultView should wait RefreshThread exi…
flinkbot commented on pull request #15520: URL: https://github.com/apache/flink/pull/15520#issuecomment-815411465 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 41dd8818b80689f45e6b24f5e7ad4edbe7b7ecf7 (Thu Apr 08 03:06:06 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22097) testChangelogResultViewClearEmptyResult fail.
[
https://issues.apache.org/jira/browse/FLINK-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-22097:
---
Labels: pull-request-available test-stability (was: test-stability)
> testChangelogResultViewClearEmptyResult fail.
> -
>
> Key: FLINK-22097
> URL: https://issues.apache.org/jira/browse/FLINK-22097
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Assignee: Shengkai Fang
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15968=logs=b2f046ab-ae17-5406-acdc-240be7e870e4=93e5ae06-d194-513d-ba8d-150ef6da1d7c=8539
> {code:java}
> Exception in thread "Thread-9" java.lang.NullPointerException
> at
> org.apache.flink.table.client.cli.CliClient.isPlainTerminal(CliClient.java:181)
> at
> org.apache.flink.table.client.cli.CliClient.clearTerminal(CliClient.java:169)
> at org.apache.flink.table.client.cli.CliView.display(CliView.java:191)
> at
> org.apache.flink.table.client.cli.CliChangelogResultView.display(CliChangelogResultView.java:101)
> at
> org.apache.flink.table.client.cli.CliResultView$RefreshThread.run(CliResultView.java:267)
> {code}
> {code:java}
> java.lang.AssertionError: Invalid number of cancellations.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.flink.table.client.cli.CliResultViewTest.testResultViewClearResult(CliResultViewTest.java:117)
> at
> org.apache.flink.table.client.cli.CliResultViewTest.testChangelogResultViewClearEmptyResult(CliResultViewTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kezhuw commented on pull request #15507: [FLINK-21386][datastream] Postpone FromElementsFunction serialization to respect later type customization
kezhuw commented on pull request #15507: URL: https://github.com/apache/flink/pull/15507#issuecomment-815410555 Test was cancelled in [kafka_gelly](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16149=logs=c5f0071e-1851-543e-9a45-9ac140befc32). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 opened a new pull request #15520: [FLINK-22097][sql-client] CliResultView should wait RefreshThread exi…
fsk119 opened a new pull request #15520: URL: https://github.com/apache/flink/pull/15520 …ts before exits ## What is the purpose of the change *The failed test is mainly to test the view thread has the ability to exit gracefully when users(main thread) interrupt the view thread.* *We have 3 threads in the test: main thread, view thread and refresh thread.* *The main thread has the resource Terminal and uses the the resource to create the view thread. When the view thread starts, it set the interrupt flag on the view thread and close the Terminal until view thread exits* - *When view thread start, it start the refresh thread and monitors the user input. In the test, it only monitors interrput signal.* - *When get the signal, it mark the flag isRunning of the refresh thread false and notify the refresh thread.* - *The refresh thread is used to fetch the data from the remote periodically and display the results on Terminal. When exits, the refresh thread will invoke the `Executor.cancelQuery` and count down the cancellation.* *The reason why we can NPE is because when view thread exits, it only notifies the refresh thread instead of waiting for the refresh thread exits. It's possible* 1. *the view thread notifies the refresh thread* 2. *the refresh thread wakes up and prepares to display the result on the terminal* 3. *the view thread exits* 4. *the main thread finds the view thread exit and closes the resource* 5. *the refresh thread gets NPE* 6. *the main thread find the cancellation is not as same as the expected* *Therefore, we should request the view thread wait for the refresh thread exit.* *To reproduce the bug, we can request the refresh thread sleep 1 seconds before display.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15506: [FLINK-21240][jdbc] Fix JDBC row converter doesn't support external LocalDateTime type
flinkbot edited a comment on pull request #15506: URL: https://github.com/apache/flink/pull/15506#issuecomment-814806856 ## CI report: * eb9027ad7da7cf563e1a051ba4301c03225f9959 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16170) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16138) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15519: [FLINK-22120] Remove duplicate code in generated code for map get
flinkbot commented on pull request #15519: URL: https://github.com/apache/flink/pull/15519#issuecomment-815409179 ## CI report: * 41890dd12eec1ca6aa45867999a5fb6d1895cd9b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15498: [FLINK-22121][table-planner-blink] FlinkLogicalRankRuleBase now check if name of rankNumberType already exists in the input
flinkbot edited a comment on pull request #15498: URL: https://github.com/apache/flink/pull/15498#issuecomment-814575824 ## CI report: * c6fc2d4fa4d1149f364432891223779d23a8f6a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16103) * ac009d9c49b7442cedee7050b503cf9164ee2683 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15489: [FLINK-22106][table-planner-blink] Result type of GeneratedExpression in StringCallGen should be compatible with their definition in
flinkbot edited a comment on pull request #15489: URL: https://github.com/apache/flink/pull/15489#issuecomment-813835360 ## CI report: * 0bd5b6f241ae6168c0abb82873f4d47448e1330c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16110) * 71bb2d2f86ca53b19dbf181ecdaa89ab4e83d44e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22146) Migrate StateBackend related Chinese docs to latest state backend
Yun Tang created FLINK-22146: Summary: Migrate StateBackend related Chinese docs to latest state backend Key: FLINK-22146 URL: https://issues.apache.org/jira/browse/FLINK-22146 Project: Flink Issue Type: Sub-task Components: Documentation, Runtime / State Backends Reporter: Yun Tang Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15485: [FLINK-20387][table] Support TIMESTAMP_LTZ as rowtime attribute
flinkbot edited a comment on pull request #15485: URL: https://github.com/apache/flink/pull/15485#issuecomment-812897189 ## CI report: * ee69df73cd5a7e61efa44b704eb5d551db23d7e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16137) * 08701416b199b4b8620240a27a4695919960ef46 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16169) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22097) testChangelogResultViewClearEmptyResult fail.
[
https://issues.apache.org/jira/browse/FLINK-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316830#comment-17316830
]
Shengkai Fang commented on FLINK-22097:
---
The failed test is mainly to test the view thread has the ability to exit
gracefully when users(main thread) interrupt the view thread.
We have 3 threads in the test: main thread, view thread and refresh thread.
- The main thread has the resource {{Terminal}} and uses the the resource to
create the view thread. When the view thread starts, it set the interrupt flag
on the view thread and *close* the {{Terminal}} until view thread exits
- When view thread start, it start the refresh thread and monitors the user
input. In the test, it only monitors interrput signal. When get the signal, it
mark the flag {{isRunning}} of the refresh thread {{false}} and notify the
refresh thread.
- The refresh thread is used to fetch the data from the remote periodically
and display the results on {{Terminal}}. When exits, the refresh thread will
invoke the \{{Executor.cancelQuery}} and count down the {{cancellation}}.
The reason why we can NPE is because when view thread exits, it only notifies
the refresh thread instead of waiting for the refresh thread exits. It's
possible
# the view thread notifies the refresh thread
# the refresh thread wakes up and prepares to display the result on the
terminal
# the view thread exits
# the main thread finds the view thread exit and closes the resource
# the refresh thread gets NPE
# the main thread find the {{cancellation}} is not as same as the expected
Therefore, we should request the view thread wait for the refresh thread exit.
To reproduce the bug, we can request the refresh thread sleep 1 seconds before
display.
> testChangelogResultViewClearEmptyResult fail.
> -
>
> Key: FLINK-22097
> URL: https://issues.apache.org/jira/browse/FLINK-22097
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Assignee: Shengkai Fang
>Priority: Major
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15968=logs=b2f046ab-ae17-5406-acdc-240be7e870e4=93e5ae06-d194-513d-ba8d-150ef6da1d7c=8539
> {code:java}
> Exception in thread "Thread-9" java.lang.NullPointerException
> at
> org.apache.flink.table.client.cli.CliClient.isPlainTerminal(CliClient.java:181)
> at
> org.apache.flink.table.client.cli.CliClient.clearTerminal(CliClient.java:169)
> at org.apache.flink.table.client.cli.CliView.display(CliView.java:191)
> at
> org.apache.flink.table.client.cli.CliChangelogResultView.display(CliChangelogResultView.java:101)
> at
> org.apache.flink.table.client.cli.CliResultView$RefreshThread.run(CliResultView.java:267)
> {code}
> {code:java}
> java.lang.AssertionError: Invalid number of cancellations.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.flink.table.client.cli.CliResultViewTest.testResultViewClearResult(CliResultViewTest.java:117)
> at
> org.apache.flink.table.client.cli.CliResultViewTest.testChangelogResultViewClearEmptyResult(CliResultViewTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
>
[jira] [Commented] (FLINK-21694) Increase default value of "state.backend.rocksdb.checkpoint.transfer.thread.num"
[ https://issues.apache.org/jira/browse/FLINK-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316829#comment-17316829 ] Yun Tang commented on FLINK-21694: -- This ticket has not been resolved in Flink-1.13. And from my points of view, this value deserves to increase. I prefer to be more cautious to increase the default value to 4 instead of 8 to avoid too much pressure to DFS during resume checkpoints. What's your ideas, [~sewen] [~liyu] > Increase default value of > "state.backend.rocksdb.checkpoint.transfer.thread.num" > > > Key: FLINK-21694 > URL: https://issues.apache.org/jira/browse/FLINK-21694 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Priority: Critical > Fix For: 1.13.0 > > > The default value for the number of threads used to download state artifacts > from checkpoint storage should be increased. > The increase should not pose risk of regression, but does in many cases speed > up checkpoint recovery significantly. > Something similar was reported in this blog post, item (3). > https://engineering.contentsquare.com/2021/ten-flink-gotchas/ > A default value of 8 (eight) sounds like a good default. It should not result > in excessive thread explosion, and already speeds up recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
flinkbot edited a comment on pull request #15518: URL: https://github.com/apache/flink/pull/15518#issuecomment-815397264 ## CI report: * aa7327706869d9cfc9c3c0a5084c9c7676dba6b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16171) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15519: [FLINK-22120] Remove duplicate code in generated code for map get
flinkbot commented on pull request #15519: URL: https://github.com/apache/flink/pull/15519#issuecomment-815402463 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 41890dd12eec1ca6aa45867999a5fb6d1895cd9b (Thu Apr 08 02:36:44 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22120) Remove duplicate code in generated code for map get
[
https://issues.apache.org/jira/browse/FLINK-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-22120:
---
Labels: pull-request-available (was: )
> Remove duplicate code in generated code for map get
> ---
>
> Key: FLINK-22120
> URL: https://issues.apache.org/jira/browse/FLINK-22120
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Zou
>Assignee: Zou
>Priority: Major
> Labels: pull-request-available
>
> There is a bug when we get values from a map with a complex expression key.
> The reason is the `key.code` in ScalarOperatorGens#generateMapGet will be
> called twice.
>
> This is a test case:
> {code:java}
> CREATE TABLE T (
> `name` VARCHAR,
> `map` MAP
> ) WITH (
> ...
> );
> SELECT `map`[coalesce(name, '')] FROM T;
> {code}
> An exception will be thrown while running this query:
> {code:java}
> Caused by: java.lang.RuntimeException: Could not instantiate generated class
> 'StreamExecCalc$21'
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
> at
> org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:167)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:458)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:527)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:725)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:549)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:65)
> ... 8 more
> Caused by:
> org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)
> ... 10 more
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table
> program cannot be compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 13 more
> Caused by: org.codehaus.commons.compiler.CompileException: Line 121, Column
> 27: Redefinition of local variable "result$6"
> at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124)
> at
> org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3662)
> at org.codehaus.janino.UnitCompiler.access$5800(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$12.visitLocalVariableDeclarationStatement(UnitCompiler.java:3543)
> at
> org.codehaus.janino.UnitCompiler$12.visitLocalVariableDeclarationStatement(UnitCompiler.java:3511)
> at
> org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3511)
> at
> org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3510)
> at
>
[jira] [Updated] (FLINK-21736) Introduce latency tracking state
[
https://issues.apache.org/jira/browse/FLINK-21736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang updated FLINK-21736:
-
Release Note: State access latency metrics are introduced to track all
kinds of keyed state access to help debug state performance. This feature is
not enabled by default and could be turned on once setting
`state.backend.latency-track.keyed-state-enabled` as true.
> Introduce latency tracking state
>
>
> Key: FLINK-21736
> URL: https://issues.apache.org/jira/browse/FLINK-21736
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> This ticket is extended from FLINK-16444 which wants to track latency of
> RocksDB options. However previous ticket has two problems:
> # Cannot apply to heap based state backend and other user customized state
> backend.
> # User need background knowledge to know what RocksDB operation
> corresponding to what state operation, e.g RocksDB adopts {{merge}} to act as
> listState#add.
> Thus, I prefer to introduce latency tracking state wrapper to detect the
> access latency. And to not impact the original state access, I would still
> use sample interval as previous FLINK-16444 did.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zoudan opened a new pull request #15519: [FLINK-22120] Remove duplicate code in generated code for map get
zoudan opened a new pull request #15519: URL: https://github.com/apache/flink/pull/15519 ## What is the purpose of the change This pull request fixed a bug that there is duplicate code in generated code for map get, an it may lead to some exceptions such as 'Redefinition of local variable'. ## Brief change log remove the duplicate code in generated code for map get. ## Verifying this change This change added tests and can be verified as follows: - org.apache.flink.table.planner.runtime.stream.sql.CalcITCase#testMapGet ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21726) Fix checkpoint stuck
[
https://issues.apache.org/jira/browse/FLINK-21726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang updated FLINK-21726:
-
Fix Version/s: (was: 1.13.0)
1.14.0
> Fix checkpoint stuck
>
>
> Key: FLINK-21726
> URL: https://issues.apache.org/jira/browse/FLINK-21726
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: fanrui
>Priority: Critical
> Fix For: 1.14.0
>
>
> h1. 1. Bug description:
> When RocksDB Checkpoint, it may be stuck in
> `WaitUntilFlushWouldNotStallWrites` method.
> h1. 2. Simple analysis of the reasons:
> h2. 2.1 Configuration parameters:
>
> {code:java}
> # Flink yaml:
> state.backend.rocksdb.predefined-options: SPINNING_DISK_OPTIMIZED_HIGH_MEM
> state.backend.rocksdb.compaction.style: UNIVERSAL
> # corresponding RocksDB config
> Compaction Style : Universal
> max_write_buffer_number : 4
> min_write_buffer_number_to_merge : 3{code}
> Checkpoint is usually very fast. When the Checkpoint is executed,
> `WaitUntilFlushWouldNotStallWrites` is called. If there are 2 Immutable
> MemTables, which are less than `min_write_buffer_number_to_merge`, they will
> not be flushed. But will enter this code.
>
> {code:java}
> // method: GetWriteStallConditionAndCause
> if (mutable_cf_options.max_write_buffer_number> 3 &&
> num_unflushed_memtables >=
> mutable_cf_options.max_write_buffer_number-1) {
> return {WriteStallCondition::kDelayed, WriteStallCause::kMemtableLimit};
> }
> {code}
> code link:
> [https://github.com/facebook/rocksdb/blob/fbed72f03c3d9e4fdca3e5993587ef2559ba6ab9/db/column_family.cc#L847]
> Checkpoint thought there was a FlushJob, but it didn't. So will always wait.
> h2. 2.2 solution:
> Increase the restriction: the `number of Immutable MemTable` >=
> `min_write_buffer_number_to_merge will wait`.
> The rocksdb community has fixed this bug, link:
> [https://github.com/facebook/rocksdb/pull/7921]
> h2. 2.3 Code that can reproduce the bug:
> [https://github.com/1996fanrui/fanrui-learning/blob/flink-1.12/module-java/src/main/java/com/dream/rocksdb/RocksDBCheckpointStuck.java]
> h1. 3. Interesting point
> This bug will be triggered only when `the number of sorted runs >=
> level0_file_num_compaction_trigger`.
> Because there is a break in WaitUntilFlushWouldNotStallWrites.
> {code:java}
> if (cfd->imm()->NumNotFlushed() <
> cfd->ioptions()->min_write_buffer_number_to_merge &&
> vstorage->l0_delay_trigger_count() <
> mutable_cf_options.level0_file_num_compaction_trigger) {
> break;
> }
> {code}
> code link:
> [https://github.com/facebook/rocksdb/blob/fbed72f03c3d9e4fdca3e5993587ef2559ba6ab9/db/db_impl/db_impl_compaction_flush.cc#L1974]
> Universal may have `l0_delay_trigger_count() >=
> level0_file_num_compaction_trigger`, so this bug is triggered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21726) Fix checkpoint stuck
[
https://issues.apache.org/jira/browse/FLINK-21726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316828#comment-17316828
]
Yun Tang commented on FLINK-21726:
--
Since this fix depends on releasing another FRocksDB version, and this bug
occurs with specific configurations. From my point of view, I plan to make the
fix version as Flink-1.14 with bumping RocksDB to latest stable version at that
time.
> Fix checkpoint stuck
>
>
> Key: FLINK-21726
> URL: https://issues.apache.org/jira/browse/FLINK-21726
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: fanrui
>Priority: Critical
> Fix For: 1.13.0
>
>
> h1. 1. Bug description:
> When RocksDB Checkpoint, it may be stuck in
> `WaitUntilFlushWouldNotStallWrites` method.
> h1. 2. Simple analysis of the reasons:
> h2. 2.1 Configuration parameters:
>
> {code:java}
> # Flink yaml:
> state.backend.rocksdb.predefined-options: SPINNING_DISK_OPTIMIZED_HIGH_MEM
> state.backend.rocksdb.compaction.style: UNIVERSAL
> # corresponding RocksDB config
> Compaction Style : Universal
> max_write_buffer_number : 4
> min_write_buffer_number_to_merge : 3{code}
> Checkpoint is usually very fast. When the Checkpoint is executed,
> `WaitUntilFlushWouldNotStallWrites` is called. If there are 2 Immutable
> MemTables, which are less than `min_write_buffer_number_to_merge`, they will
> not be flushed. But will enter this code.
>
> {code:java}
> // method: GetWriteStallConditionAndCause
> if (mutable_cf_options.max_write_buffer_number> 3 &&
> num_unflushed_memtables >=
> mutable_cf_options.max_write_buffer_number-1) {
> return {WriteStallCondition::kDelayed, WriteStallCause::kMemtableLimit};
> }
> {code}
> code link:
> [https://github.com/facebook/rocksdb/blob/fbed72f03c3d9e4fdca3e5993587ef2559ba6ab9/db/column_family.cc#L847]
> Checkpoint thought there was a FlushJob, but it didn't. So will always wait.
> h2. 2.2 solution:
> Increase the restriction: the `number of Immutable MemTable` >=
> `min_write_buffer_number_to_merge will wait`.
> The rocksdb community has fixed this bug, link:
> [https://github.com/facebook/rocksdb/pull/7921]
> h2. 2.3 Code that can reproduce the bug:
> [https://github.com/1996fanrui/fanrui-learning/blob/flink-1.12/module-java/src/main/java/com/dream/rocksdb/RocksDBCheckpointStuck.java]
> h1. 3. Interesting point
> This bug will be triggered only when `the number of sorted runs >=
> level0_file_num_compaction_trigger`.
> Because there is a break in WaitUntilFlushWouldNotStallWrites.
> {code:java}
> if (cfd->imm()->NumNotFlushed() <
> cfd->ioptions()->min_write_buffer_number_to_merge &&
> vstorage->l0_delay_trigger_count() <
> mutable_cf_options.level0_file_num_compaction_trigger) {
> break;
> }
> {code}
> code link:
> [https://github.com/facebook/rocksdb/blob/fbed72f03c3d9e4fdca3e5993587ef2559ba6ab9/db/db_impl/db_impl_compaction_flush.cc#L1974]
> Universal may have `l0_delay_trigger_count() >=
> level0_file_num_compaction_trigger`, so this bug is triggered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
flinkbot commented on pull request #15518: URL: https://github.com/apache/flink/pull/15518#issuecomment-815397264 ## CI report: * aa7327706869d9cfc9c3c0a5084c9c7676dba6b5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15506: [FLINK-21240][jdbc] Fix JDBC row converter doesn't support external LocalDateTime type
flinkbot edited a comment on pull request #15506: URL: https://github.com/apache/flink/pull/15506#issuecomment-814806856 ## CI report: * eb9027ad7da7cf563e1a051ba4301c03225f9959 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16138) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16170) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15485: [FLINK-20387][table] Support TIMESTAMP_LTZ as rowtime attribute
flinkbot edited a comment on pull request #15485: URL: https://github.com/apache/flink/pull/15485#issuecomment-812897189 ## CI report: * ee69df73cd5a7e61efa44b704eb5d551db23d7e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16137) * 08701416b199b4b8620240a27a4695919960ef46 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22145) CheckStyle for scala not work
MaChengLong created FLINK-22145: --- Summary: CheckStyle for scala not work Key: FLINK-22145 URL: https://issues.apache.org/jira/browse/FLINK-22145 Project: Flink Issue Type: Improvement Components: API / Scala Reporter: MaChengLong I followed this doc [https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/flinkdev/ide_setup/] to setup scala code style format but when i format exists scala code with code->reformat code some original scala code style was broken,it seems CheckStyle for scala( tools/maven/scalastyle-config.xml was placed to .idea/) not work -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16908) FlinkKafkaProducerITCase testScaleUpAfterScalingDown Timeout expired while initializing transactional state in 60000ms.
[
https://issues.apache.org/jira/browse/FLINK-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316819#comment-17316819
]
Dong Lin commented on FLINK-16908:
--
I think this is due to the same issue as described in
https://issues.apache.org/jira/browse/FLINK-18634?focusedCommentId=17284445=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17284445
The solution is to bump up Kafka dependency to 2.5.1.
However, Kafka 2.5.1 has dropped support for Scala 2.11. Flink will also need
to drop support for Scala 2.11 in order to bump up the Kafka dependency to
2.5.1. More discussion/work is needed in order to make this change.
> FlinkKafkaProducerITCase testScaleUpAfterScalingDown Timeout expired while
> initializing transactional state in 6ms.
> ---
>
> Key: FLINK-16908
> URL: https://issues.apache.org/jira/browse/FLINK-16908
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Piotr Nowojski
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6889=logs=c5f0071e-1851-543e-9a45-9ac140befc32=f66652e3-384e-5b25-be29-abfea69ea8da
> {noformat}
> [ERROR]
> testScaleUpAfterScalingDown(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 64.353 s <<< ERROR!
> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
> initializing transactional state in 6ms.
> {noformat}
> After this initial error many other tests (I think all following unit tests)
> failed with errors like:
> {noformat}
> [ERROR]
> testFailAndRecoverSameCheckpointTwice(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 7.895 s <<< FAILURE!
> java.lang.AssertionError: Detected producer leak. Thread name:
> kafka-producer-network-thread | producer-196
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.checkProducerLeak(FlinkKafkaProducerITCase.java:675)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testFailAndRecoverSameCheckpointTwice(FlinkKafkaProducerITCase.java:311)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] leonardBang commented on pull request #15506: [FLINK-21240][jdbc] Fix JDBC row converter doesn't support external LocalDateTime type
leonardBang commented on pull request #15506: URL: https://github.com/apache/flink/pull/15506#issuecomment-815385594 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18634) FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout expired after 60000milliseconds while awaiting InitProducerId"
[
https://issues.apache.org/jira/browse/FLINK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316817#comment-17316817
]
Dong Lin commented on FLINK-18634:
--
[~trohrmann] It appears that flink-tests/src/test's root logger level is set to
INFO. I created [https://github.com/apache/flink/pull/15518] to set its log4j
level to OFF. I am waiting to see whether this change could fix the issue.
> FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout
> expired after 6milliseconds while awaiting InitProducerId"
>
>
> Key: FLINK-18634
> URL: https://issues.apache.org/jira/browse/FLINK-18634
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0, 1.13.0
>Reporter: Dian Fu
>Assignee: Jiangjie Qin
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4590=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-07-17T11:43:47.9693015Z [ERROR] Tests run: 12, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 269.399 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> 2020-07-17T11:43:47.9693862Z [ERROR]
> testRecoverCommittedTransaction(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 60.679 s <<< ERROR!
> 2020-07-17T11:43:47.9694737Z org.apache.kafka.common.errors.TimeoutException:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> 2020-07-17T11:43:47.9695376Z Caused by:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22063) Lookup Join outputs wrong results for some scenario
[
https://issues.apache.org/jira/browse/FLINK-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316816#comment-17316816
]
Leonard Xu commented on FLINK-22063:
Hi, [~icshuo] Do you have any progress?
> Lookup Join outputs wrong results for some scenario
> ---
>
> Key: FLINK-22063
> URL: https://issues.apache.org/jira/browse/FLINK-22063
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.2
>Reporter: Shuo Cheng
>Assignee: Shuo Cheng
>Priority: Critical
> Fix For: 1.13.0
>
>
> Reproduce the bug as following:
> In LookupJoinITCase, given the sql
> {code:sql}
> SELECT
> T.id, T.len, D.id, T.content, D.name
> FROM src AS T JOIN user_table for system_time as of T.proctime AS D
> ON T.id = D.id and cast(T.len as bigint) = D.id
> {code}
> the following execution plan is generated:
> {code:java}
> LegacySink(name=[DataStreamTableSink], fields=[id, len, id0, content, name])
> +- Calc(select=[id, len, id0, content, name])
>+- LookupJoin(table=[**], joinType=[InnerJoin], async=[false],
> lookup=[id=len0], select=[id, len, content, len0, id, name])
> +- Calc(select=[id, len, content, CAST(len) AS len0])
> +- TableSourceScan(table=[[**]], fields=[id, len, content])
> {code}
> As we can see, the condition `T.id = D.id` is lost, so a wrong result may be
> produced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
flinkbot commented on pull request #15518: URL: https://github.com/apache/flink/pull/15518#issuecomment-815384604 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit aa7327706869d9cfc9c3c0a5084c9c7676dba6b5 (Thu Apr 08 01:42:15 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 opened a new pull request #15518: [hotfix][tests] Set flink-tests/src/test root logger level to OFF
lindong28 opened a new pull request #15518: URL: https://github.com/apache/flink/pull/15518 ## Contribution Checklist This PR sets flink-tests/src/test root logger level to OFF. This simplifies the parsing of Azure Pipeline logs by not flooding Azure Pipeline logs with INFO/WARN messages. ## Brief change log This PR sets flink-tests/src/test root logger level to OFF. ## Verifying this change To be done in Azure pipeline execution. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19159) Using Scalafmt to format scala source code
[ https://issues.apache.org/jira/browse/FLINK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316811#comment-17316811 ] MaChengLong commented on FLINK-19159: - I followed this docs [https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/flinkdev/ide_setup/] to setup checkstyle for scala but willn't work with my intellij idea 2020.1 or 2021.1 (when i format any exists scala code using code->reformat code,the original format was broken), so,what's the current way to format scala source code? > Using Scalafmt to format scala source code > -- > > Key: FLINK-19159 > URL: https://issues.apache.org/jira/browse/FLINK-19159 > Project: Flink > Issue Type: Improvement > Components: API / Scala >Reporter: darion yaphet >Priority: Minor > > Scalafmt is a code formatter for Scala. It can help developer to avoid code > style conflict -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20431) KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 expected:<10> but was:<1>
[ https://issues.apache.org/jira/browse/FLINK-20431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated FLINK-20431: - Fix Version/s: (was: 1.13.0) 1.13.1 > KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 > expected:<10> but was:<1> > - > > Key: FLINK-20431 > URL: https://issues.apache.org/jira/browse/FLINK-20431 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.13.0 >Reporter: Huang Xingbo >Assignee: Jiangjie Qin >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.12.3, 1.13.1 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10351=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > [ERROR] Failures: > [ERROR] > KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 > expected:<10> but was:<1> > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15493) FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator failed on travis
[
https://issues.apache.org/jira/browse/FLINK-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin updated FLINK-15493:
-
Fix Version/s: (was: 1.13.0)
1.13.1
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis
> ---
>
> Key: FLINK-15493
> URL: https://issues.apache.org/jira/browse/FLINK-15493
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.10.0, 1.13.0
>Reporter: Dian Fu
>Assignee: Jiangjie Qin
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.1
>
>
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis with the following exception:
> {code}
> Test
> testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase)
> failed with: org.junit.runners.model.TestTimedOutException: test timed out
> after 3 milliseconds at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:177)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:115)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:197)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(FlinkKafkaInternalProducerITCase.java:176)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> instance: [https://api.travis-ci.org/v3/job/633307060/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22075) Incorrect null outputs in left join
[
https://issues.apache.org/jira/browse/FLINK-22075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316796#comment-17316796
]
Jamie Brandon edited comment on FLINK-22075 at 4/8/21, 12:01 AM:
-
In batch mode with a file source the results are actually worse. Only one
non-null row was returned.
https://github.com/jamii/streaming-consistency/commit/699fc88bf34886063777efca1e57ba0221302ee1
{code:bash}
jamie@machine:~/streaming-consistency/flink$ wc -l tmp/outer_join_with_time
100 tmp/outer_join_with_time
jamie@machine:~/streaming-consistency/flink$ grep -v null
tmp/outer_join_with_time
insert 32896,32896
{code}
was (Author: jamii):
In batch mode with a file source the results are actually worse. Only one
non-null row was returned.
https://github.com/jamii/streaming-consistency/compare/flink-batch-mode
{code:bash}
jamie@machine:~/streaming-consistency/flink$ wc -l tmp/outer_join_with_time
100 tmp/outer_join_with_time
jamie@machine:~/streaming-consistency/flink$ grep -v null
tmp/outer_join_with_time
insert 32896,32896
{code}
> Incorrect null outputs in left join
> ---
>
> Key: FLINK-22075
> URL: https://issues.apache.org/jira/browse/FLINK-22075
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.2
> Environment:
> https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/pkgs.nix#L25-L46
> ```
> [nix-shell:~/streaming-consistency/flink]$ java -version
> openjdk version "1.8.0_265"
> OpenJDK Runtime Environment (build 1.8.0_265-ga)
> OpenJDK 64-Bit Server VM (build 25.265-bga, mixed mode)
> [nix-shell:~/streaming-consistency/flink]$ flink --version
> Version: 1.12.2, Commit ID: 4dedee0
> [nix-shell:~/streaming-consistency/flink]$ nix-info
> system: "x86_64-linux", multi-user?: yes, version: nix-env (Nix) 2.3.10,
> channels(jamie): "", channels(root): "nixos-20.09.3554.f8929dce13e", nixpkgs:
> /nix/var/nix/profiles/per-user/root/channels/nixos
> ```
>Reporter: Jamie Brandon
>Assignee: Jingsong Lee
>Priority: Critical
> Fix For: 1.13.0
>
>
> I'm left joining a table with itself
> [here](https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/flink/src/main/java/Demo.java#L55-L66).
> The output should have no nulls, or at least emit nulls and then retract
> them. Instead I see:
> ```
> jamie@machine:~/streaming-consistency/flink$ wc -l tmp/outer_join_with_time
> 10 tmp/outer_join_with_time
> jamie@machine:~/streaming-consistency/flink$ grep -c insert
> tmp/outer_join_with_time
> 10
> jamie@machine:~/streaming-consistency/flink$ grep -c 'null'
> tmp/outer_join_with_time
> 16943
> ```
> ~17% of the outputs are incorrect and never retracted.
> [Full
> output](https://gist.githubusercontent.com/jamii/983fee41609b1425fe7fa59d3249b249/raw/069b9dcd4faf9f6113114381bc7028c6642ca787/gistfile1.txt)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22075) Incorrect null outputs in left join
[
https://issues.apache.org/jira/browse/FLINK-22075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316796#comment-17316796
]
Jamie Brandon commented on FLINK-22075:
---
In batch mode with a file source the results are actually worse. Only one
non-null row was returned.
https://github.com/jamii/streaming-consistency/compare/flink-batch-mode
{code:bash}
jamie@machine:~/streaming-consistency/flink$ wc -l tmp/outer_join_with_time
100 tmp/outer_join_with_time
jamie@machine:~/streaming-consistency/flink$ grep -v null
tmp/outer_join_with_time
insert 32896,32896
{code}
> Incorrect null outputs in left join
> ---
>
> Key: FLINK-22075
> URL: https://issues.apache.org/jira/browse/FLINK-22075
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.2
> Environment:
> https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/pkgs.nix#L25-L46
> ```
> [nix-shell:~/streaming-consistency/flink]$ java -version
> openjdk version "1.8.0_265"
> OpenJDK Runtime Environment (build 1.8.0_265-ga)
> OpenJDK 64-Bit Server VM (build 25.265-bga, mixed mode)
> [nix-shell:~/streaming-consistency/flink]$ flink --version
> Version: 1.12.2, Commit ID: 4dedee0
> [nix-shell:~/streaming-consistency/flink]$ nix-info
> system: "x86_64-linux", multi-user?: yes, version: nix-env (Nix) 2.3.10,
> channels(jamie): "", channels(root): "nixos-20.09.3554.f8929dce13e", nixpkgs:
> /nix/var/nix/profiles/per-user/root/channels/nixos
> ```
>Reporter: Jamie Brandon
>Assignee: Jingsong Lee
>Priority: Critical
> Fix For: 1.13.0
>
>
> I'm left joining a table with itself
> [here](https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/flink/src/main/java/Demo.java#L55-L66).
> The output should have no nulls, or at least emit nulls and then retract
> them. Instead I see:
> ```
> jamie@machine:~/streaming-consistency/flink$ wc -l tmp/outer_join_with_time
> 10 tmp/outer_join_with_time
> jamie@machine:~/streaming-consistency/flink$ grep -c insert
> tmp/outer_join_with_time
> 10
> jamie@machine:~/streaming-consistency/flink$ grep -c 'null'
> tmp/outer_join_with_time
> 16943
> ```
> ~17% of the outputs are incorrect and never retracted.
> [Full
> output](https://gist.githubusercontent.com/jamii/983fee41609b1425fe7fa59d3249b249/raw/069b9dcd4faf9f6113114381bc7028c6642ca787/gistfile1.txt)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol commented on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
zentol commented on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-815341717 > While it fixes the RescalingITCase , it also introduces the issue that we may use a smaller maxParallelism than we initially used for requesting resources, when the AdaptiveScheduler is used without reactive mode. So even if the max parallelism was set to 8 based on the savepoint information, the scheduler will still initially ask for 128+ slots, and hold on to them until the job terminates. Thinking back on it, this may be incorrect (duh). With reactive mode we reject this case because we used a higher max paralllelism than what was set in the savepoint. Without reactive mode, we define the initial requirements based on the parallelism, not max parallelism. So long as the parallelism does not exceed the initially derived or set max parallelism the job will run fine without wasting resources. If it exceeds the max parallelism set in the savepoint then the job will fail, which is fine because that are the semantics of the max parallelism. God this issue is messing with my brain... 勞 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] austince edited a comment on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
austince edited a comment on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-815329177 > Unless I'm missing anything, then here is an example where this happens: > > ``` > P1= 80 => MP1=128 > P2=100 => MP2=256 > ``` > > So similarly to option 2, with option 1 we still have this inconsistency that can very well break existing jobs when migrated to the adaptive scheduler, _or at some point in the future after migration_. The only way to prevent that is option 3, or, option 4: outright reject jobs that have not explicitly set the max parallelism. That is possible, I just created a test case that proves it. So, I think option 4 (require max parallelism to be set) would be the simplest to get in, and not a difficult constraint to communicate to users because a) Adaptive scheduler + Reactive Mode are new and "experimental" features, b) setting max parallelism on all operators is already documented as a best practice for production jobs, and c) there is a solid solution that can immediately be queued up for the next release (reading savepoints before creating the graph). I guess something @tillrohrmann + @knaufk (original ticket author, FLINK-21844) should weigh in on? I think option 3 would be a temporary solution and would get tricky, as there is no communication between the scheduler and the state restore at the moment, and there are quite a few layers in between. Unless I misunderstand the necessary updates for that option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] austince edited a comment on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
austince edited a comment on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-815329177 > Unless I'm missing anything, then here is an example where this happens: > > ``` > P1= 80 => MP1=128 > P2=100 => MP2=256 > ``` > > So similarly to option 2, with option 1 we still have this inconsistency that can very well break existing jobs when migrated to the adaptive scheduler, _or at some point in the future after migration_. The only way to prevent that is option 3, or, option 4: outright reject jobs that have not explicitly set the max parallelism. That is possible, I just created a test case that proves it. So, I think option 4 would be the simplest to get in, and not a difficult constraint to communicate to users because a) Adaptive scheduler + Reactive Mode are new and "experimental" features, b) setting max parallelism on all operators is already documented as a best practice for production jobs, and c) there is a solid solution that can immediately be queued up for the next release (reading savepoints before creating the graph). I guess something @tillrohrmann + @knaufk (original ticket author, FLINK-21844) should weigh in on? I think option 3 would be a temporary solution and would get tricky, as there is no communication between the scheduler and the state restore at the moment, and there are quite a few layers in between. Unless I misunderstand the necessary updates for that option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
flinkbot edited a comment on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-814453955 ## CI report: * eaf1f344b8e82f4afc1a0d96c3a5602011f05ff1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16158) * f39c534bfa814d33dbe0646fe79c09f5b5cc32a9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16164) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15496: [FLINK-21990][streaming] Cancel task before clean up if execution was…
flinkbot edited a comment on pull request #15496: URL: https://github.com/apache/flink/pull/15496#issuecomment-814224240 ## CI report: * c756b734d8117d19d9b825b70e6475cd3bcfa4d5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16156) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] austince edited a comment on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
austince edited a comment on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-815329177 > Unless I'm missing anything, then here is an example where this happens: > > ``` > P1= 80 => MP1=128 > P2=100 => MP2=256 > ``` > > So similarly to option 2, with option 1 we still have this inconsistency that can very well break existing jobs when migrated to the adaptive scheduler, _or at some point in the future after migration_. The only way to prevent that is option 3, or, option 4: outright reject jobs that have not explicitly set the max parallelism. That is possible, I just created a test case that proves it. So, I think option 4 would be the simplest to get in, and not a difficult constraint to communicate to users because a) Adaptive scheduler + Reactive Mode are new and "experimental" features, b) setting max parallelism is already documented as a best practice for production jobs, and c) there is a solid solution that can immediately be queued up for the next release (reading savepoints before creating the graph). I guess something @tillrohrmann + @knaufk (original ticket author, FLINK-21844) should weigh in on? I think option 3 would be a temporary solution and would get tricky, as there is no communication between the scheduler and the state restore at the moment, and there are quite a few layers in between. Unless I misunderstand the necessary updates for that option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
flinkbot edited a comment on pull request #15307: URL: https://github.com/apache/flink/pull/15307#issuecomment-803592550 ## CI report: * 2ef40343abecf693b39b7b7fcb9f6d0d26bf82cd UNKNOWN * f7dde849a55b4aad1044ac200b390a604029c710 UNKNOWN * a3341911ab289f00a4aa50b99c8b90980f5fe453 UNKNOWN * d0b9eaf2a807fcec1d0bb54673ba0572390b4c31 UNKNOWN * 24d902609498c22d246a2d8618da856ae3b39a2e UNKNOWN * 1a7959069b939522c2160e35c97d221c44316aa7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16159) * 23bca0d519a07b94b82676ac2c619271f4674051 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16163) * 3853944d77be88a1657f80208e82b9108e944b46 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16168) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] austince commented on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
austince commented on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-815329177 > Unless I'm missing anything, then here is an example where this happens: > > ``` > P1= 80 => MP1=128 > P2=100 => MP2=256 > ``` > > So similarly to option 2, with option 1 we still have this inconsistency that can very well break existing jobs when migrated to the adaptive scheduler, _or at some point in the future after migration_. The only way to prevent that is option 3, or, option 4: outright reject jobs that have not explicitly set the max parallelism. That is possible, I just created a test case that proves it. So, I think option 4 would be the simplest to get in, and not a difficult constraint to communicate to users because a) Adaptive scheduler + Reactive Mode are new and "experimental" features, b) setting max parallelism is already documented as a best practice for production jobs, and c) there is a solid solution that can immediately be queued up for the next release (reading savepoints before creating the graph). I guess something @tillrohrmann + @knaufk (original ticket author, FLINK-21844) should weigh in on? I think option 3 would be a temporary solution and would get tricky, as there is no good communication between the scheduler and the execution vertex at the moment. Unless I misunderstand the necessary updates for that option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #15514: [hot fix]remove the semicolon
zentol closed pull request #15514: URL: https://github.com/apache/flink/pull/15514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #15514: [hot fix]remove the semicolon
zentol commented on pull request #15514: URL: https://github.com/apache/flink/pull/15514#issuecomment-815328675 This change is incorrect; the semi-colon is required because the class is an enum; without it the class no longer compiles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15121: The method $(String) is undefined for the type TableExample
flinkbot edited a comment on pull request #15121: URL: https://github.com/apache/flink/pull/15121#issuecomment-793480240 ## CI report: * cf26ce895a7956d258acc073817f578558e78227 UNKNOWN * 709c4110370b845f42d916a06e56c6026cf2fac8 UNKNOWN * 4ea99332e1997eebca1f3f0a9d9229b8265fe32c UNKNOWN * a68101fa0d306bbc6a4d4887f3379ae27633e142 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16157) * 69cd36d9aa6e4aeb2ad827020d125712307ab585 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16167) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21643) JDBC sink should be able to execute statements on multiple tables
[ https://issues.apache.org/jira/browse/FLINK-21643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Obuchowski updated FLINK-21643: -- Affects Version/s: (was: 1.12.2) 1.13.0 > JDBC sink should be able to execute statements on multiple tables > - > > Key: FLINK-21643 > URL: https://issues.apache.org/jira/browse/FLINK-21643 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Affects Versions: 1.13.0 >Reporter: Maciej Obuchowski >Priority: Major > Labels: pull-request-available > > Currently datastream JDBC sink supports outputting data only to one table - > by having to provide SQL template, from which SimpleBatchStatementExecutor > creates PreparedStatement. Creating multiple sinks, each of which writes data > to one table is impractical for moderate to large number of tables - > relational databases don't usually tolerate large number of connections. > I propose adding DynamicBatchStatementExecutor, which will additionally > require > 1) provided mechanism to create SQL statements based on given object > 2) cache for prepared statements > 3) mechanism for determining which statement should be used for given object -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol edited a comment on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
zentol edited a comment on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-815303500 > We could also add the invariant that is currently defined for reactive mode, where the savepoint can set the max parallelism as long it is at least what has been auto-configured, though I agree that this adds yet another bit of complexity to this tiny use case. It also wouldn't solve the issue for the RescalingITCase :/ I would rule out option 2 because it will cause the system to behave inconsistently; it will work if the initially set max parallelism is larger then the one derived from the parallelism, until a user increases the parallelism too much. If we're breaking the behavior, let's be strict about it. As for 1vs3, so far we only concerned ourselves with the case where the max parallelism was set initially, and then in a later submission was removed. I'm perfectly content with removing support for such use-cases. However, I'm concerned about the following: Consider a job for which the max parallelism was never set by the user. We ran the with with some initial parallelism P1, derived max parallelism MP1, and stored it in the savepoint. On the second run we increase the parallelism to P2. Is there a case where P2 > P1 AND P2 < MP1, but the derived max parallelism MP2 > MP1? IOW, is there a case where a user can increase the parallelism such that it is still reasonable to expect it to run (because we aren't exceeding the initial max parallelism), but our new strict rules would forbid it to do so? If so, then I think we cannot really deny this from working. Unless I'm missing anything, then here is an example where this happens: ``` P1= 80 => MP1=128 P2=100 => MP2=256 ``` So similarly to option 2, with option 1 we still have this inconsistency that can very well break existing jobs when migrated to the adaptive scheduler, _or at some point in the future after migration_. The only way to prevent that is option 3, or, option 4: outright reject jobs that have not explicitly set the max parallelism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #15497: [FLINK-22084][runtime] Use a consistent default max parallelism in the Adaptive Scheduler
zentol commented on pull request #15497: URL: https://github.com/apache/flink/pull/15497#issuecomment-815303500 > We could also add the invariant that is currently defined for reactive mode, where the savepoint can set the max parallelism as long it is at least what has been auto-configured, though I agree that this adds yet another bit of complexity to this tiny use case. It also wouldn't solve the issue for the RescalingITCase :/ I would rule out option 2 because it will cause the system to behave inconsistently; it will work if the initially set max parallelism is larger then the one derived from the parallelism, until a user increases the parallelism too much. If we're breaking the behavior, let's be strict about it. As for 1vs3, so far we only concerned ourselves with the case where the max parallelism was set initially, and then in a later submission was removed. I'm perfectly content with removing support for such use-cases. However, I'm concerned about the following: Consider a job for which the max parallelism was never set by the user. We ran the with with some initial parallelism P1, derived max parallelism MP1, and stored it in the savepoint. On the second run we increase the parallelism to P2. Is there a case where P2 > p1 AND P2 < MP1, but the derived max parallelism MP2 > MP1? IOW, is there a case where a user can increase the parallelism such that it is still reasonable to expect it to run, but our new strict rules would forbid it to do so? If so, then I think we cannot really deny this from working. Unless I'm missing anything, then here is an example where this happens: ``` P1= 80 => MP1=128 P2=100 => MP2=256 ``` So similarly to option 2, with option 1 we still have this inconsistency that can very well break existing jobs when migrated to the adaptive scheduler, _or at some point in the future after migration_. The only way to prevent that is option 3, or, option 4: outright reject jobs that have not explicitly set the max parallelism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15507: [FLINK-21386][datastream] Postpone FromElementsFunction serialization to respect later type customization
flinkbot edited a comment on pull request #15507: URL: https://github.com/apache/flink/pull/15507#issuecomment-814806931 ## CI report: * 493e7604c1bfd5b5eae8df1c9db4624c89839efe Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16149) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15307: [FLINK-21675][table-planner-blink] Allow Predicate Pushdown with Watermark Assigner Between Filter and Scan
flinkbot edited a comment on pull request #15307: URL: https://github.com/apache/flink/pull/15307#issuecomment-803592550 ## CI report: * 2ef40343abecf693b39b7b7fcb9f6d0d26bf82cd UNKNOWN * f7dde849a55b4aad1044ac200b390a604029c710 UNKNOWN * a3341911ab289f00a4aa50b99c8b90980f5fe453 UNKNOWN * d0b9eaf2a807fcec1d0bb54673ba0572390b4c31 UNKNOWN * 4f11afcd0246725b6f781d22f1097affc4c3d830 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16092) * 24d902609498c22d246a2d8618da856ae3b39a2e UNKNOWN * 1a7959069b939522c2160e35c97d221c44316aa7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16159) * 23bca0d519a07b94b82676ac2c619271f4674051 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16163) * 3853944d77be88a1657f80208e82b9108e944b46 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16168) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15121: The method $(String) is undefined for the type TableExample
flinkbot edited a comment on pull request #15121: URL: https://github.com/apache/flink/pull/15121#issuecomment-793480240 ## CI report: * cf26ce895a7956d258acc073817f578558e78227 UNKNOWN * 46b4644a2a5d473b01dfb4e2a6e4b1ec3cc483d1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16121) * 709c4110370b845f42d916a06e56c6026cf2fac8 UNKNOWN * 4ea99332e1997eebca1f3f0a9d9229b8265fe32c UNKNOWN * a68101fa0d306bbc6a4d4887f3379ae27633e142 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16157) * 69cd36d9aa6e4aeb2ad827020d125712307ab585 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16167) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15121: The method $(String) is undefined for the type TableExample
flinkbot edited a comment on pull request #15121: URL: https://github.com/apache/flink/pull/15121#issuecomment-793480240 ## CI report: * cf26ce895a7956d258acc073817f578558e78227 UNKNOWN * 46b4644a2a5d473b01dfb4e2a6e4b1ec3cc483d1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16121) * 709c4110370b845f42d916a06e56c6026cf2fac8 UNKNOWN * 4ea99332e1997eebca1f3f0a9d9229b8265fe32c UNKNOWN * a68101fa0d306bbc6a4d4887f3379ae27633e142 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16157) * 69cd36d9aa6e4aeb2ad827020d125712307ab585 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21873) CoordinatedSourceRescaleITCase.testUpscaling fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-21873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316684#comment-17316684
]
Arvid Heise commented on FLINK-21873:
-
Merged into master as 9953206599910983425dceea7a48164370fa605b and 1.12 as
69cd36d9aa6e4aeb2ad827020d125712307ab585.
> CoordinatedSourceRescaleITCase.testUpscaling fails on AZP
> -
>
> Key: FLINK-21873
> URL: https://issues.apache.org/jira/browse/FLINK-21873
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Arvid Heise
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> The test {{CoordinatedSourceRescaleITCase.testUpscaling}} fails on AZP with
> {code}
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> ... 4 more
> Caused by: java.lang.Exception: successfully restored checkpoint
> at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase$FailingMapFunction.map(CoordinatedSourceRescaleITCase.java:139)
> at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase$FailingMapFunction.map(CoordinatedSourceRescaleITCase.java:126)
> at
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
> at
> org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:161)
> at
> org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
> at
> org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:101)
> at
> org.apache.flink.api.connector.source.lib.util.IteratorSourceReader.pollNext(IteratorSourceReader.java:95)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:275)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:408)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:190)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:624)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:588)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:760)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14997=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf=22049
> cc [~AHeise]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-21873) CoordinatedSourceRescaleITCase.testUpscaling fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-21873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-21873.
-
Fix Version/s: (was: 1.14.0)
1.13.0
Resolution: Fixed
> CoordinatedSourceRescaleITCase.testUpscaling fails on AZP
> -
>
> Key: FLINK-21873
> URL: https://issues.apache.org/jira/browse/FLINK-21873
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Arvid Heise
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> The test {{CoordinatedSourceRescaleITCase.testUpscaling}} fails on AZP with
> {code}
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> ... 4 more
> Caused by: java.lang.Exception: successfully restored checkpoint
> at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase$FailingMapFunction.map(CoordinatedSourceRescaleITCase.java:139)
> at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase$FailingMapFunction.map(CoordinatedSourceRescaleITCase.java:126)
> at
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
> at
> org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:161)
> at
> org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
> at
> org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:101)
> at
> org.apache.flink.api.connector.source.lib.util.IteratorSourceReader.pollNext(IteratorSourceReader.java:95)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:275)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:408)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:190)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:624)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:588)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:760)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14997=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf=22049
> cc [~AHeise]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #15503: [FLINK-21873][connector/common] Harden CoordinatedSourceRescaleITCase by checking exceptions more leniently.
AHeise merged pull request #15503: URL: https://github.com/apache/flink/pull/15503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise merged pull request #15504: [FLINK-21873][connector/common] Harden CoordinatedSourceRescaleITCase by checking exceptions more leniently.
AHeise merged pull request #15504: URL: https://github.com/apache/flink/pull/15504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
