[GitHub] [flink] twalthr commented on a change in pull request #16626: [FLINK-23498][table] Introduce a full PlannerConfiguration
twalthr commented on a change in pull request #16626: URL: https://github.com/apache/flink/pull/16626#discussion_r681493059 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java ## @@ -190,7 +190,7 @@ private final PipelineExecutorServiceLoader executorServiceLoader; -private final Configuration configuration; +protected final Configuration configuration; Review comment: I added a comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-21920) Optimize ExecutionGraphToInputsLocationsRetrieverAdapter
[
https://issues.apache.org/jira/browse/FLINK-21920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhilong Hong closed FLINK-21920.
Resolution: Later
The proposed solution will make
{{ExecutionGraphToInputsLocationsRetrieverAdapter}} stateful, which is hard to
maintain. However, the improvement of this optimization seems not obvious.
> Optimize ExecutionGraphToInputsLocationsRetrieverAdapter
>
>
> Key: FLINK-21920
> URL: https://issues.apache.org/jira/browse/FLINK-21920
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Zhilong Hong
>Priority: Major
> Labels: auto-unassigned, pull-request-available
>
> Based on the scheduler benchmark introduced in FLINK-21731, we find that
> there's a procedure related to {{DefaultScheduler#allocateSlots}} that has
> O(N^2) complexity, which is:
> {{ExecutionGraphToInputsLocationsRetrieverAdapter#getConsumedResultPartitionsProducers}}.
> The original implementation is:
> {code:java}
> for all SchedulingExecutionVertex in DefaultScheduler:
> for all ConsumedPartitionGroup of the SchedulingExecutionVertex:
> for all IntermediateResultPartition in the ConsumedPartitionGroup:
> get producer of the IntermediateResultPartition {code}
> This procedure has O(N^2) complexity.
> We can see that for each SchedulingExecutionVertex, the producers of its
> ConsumedPartitionGroup is calculated separately. For the
> SchedulingExecutionVertices in the same ConsumerVertexGroup, they have the
> same ConsumedPartitionGroup. Therefore, we don't need to calculate the
> producers over and over again. We can use a local cache to cache the
> producers. This will decrease the complexity from O(N^2) to O(N).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21920) Optimize ExecutionGraphToInputsLocationsRetrieverAdapter
[
https://issues.apache.org/jira/browse/FLINK-21920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392042#comment-17392042
]
Zhilong Hong edited comment on FLINK-21920 at 8/3/21, 7:11 AM:
---
The proposed solution will make
{{ExecutionGraphToInputsLocationsRetrieverAdapter}} stateful, which is hard to
maintain. However, the improvement of this optimization seems not obvious. For
now we just close it. If later we got a better idea, we'd like to reopen it.
was (Author: thesharing):
The proposed solution will make
{{ExecutionGraphToInputsLocationsRetrieverAdapter}} stateful, which is hard to
maintain. However, the improvement of this optimization seems not obvious.
> Optimize ExecutionGraphToInputsLocationsRetrieverAdapter
>
>
> Key: FLINK-21920
> URL: https://issues.apache.org/jira/browse/FLINK-21920
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Zhilong Hong
>Priority: Major
> Labels: auto-unassigned, pull-request-available
>
> Based on the scheduler benchmark introduced in FLINK-21731, we find that
> there's a procedure related to {{DefaultScheduler#allocateSlots}} that has
> O(N^2) complexity, which is:
> {{ExecutionGraphToInputsLocationsRetrieverAdapter#getConsumedResultPartitionsProducers}}.
> The original implementation is:
> {code:java}
> for all SchedulingExecutionVertex in DefaultScheduler:
> for all ConsumedPartitionGroup of the SchedulingExecutionVertex:
> for all IntermediateResultPartition in the ConsumedPartitionGroup:
> get producer of the IntermediateResultPartition {code}
> This procedure has O(N^2) complexity.
> We can see that for each SchedulingExecutionVertex, the producers of its
> ConsumedPartitionGroup is calculated separately. For the
> SchedulingExecutionVertices in the same ConsumerVertexGroup, they have the
> same ConsumedPartitionGroup. Therefore, we don't need to calculate the
> producers over and over again. We can use a local cache to cache the
> producers. This will decrease the complexity from O(N^2) to O(N).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23593) Performance regression on 15.07.2021
[
https://issues.apache.org/jira/browse/FLINK-23593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-23593:
---
Priority: Blocker (was: Critical)
> Performance regression on 15.07.2021
>
>
> Key: FLINK-23593
> URL: https://issues.apache.org/jira/browse/FLINK-23593
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Benchmarks
>Affects Versions: 1.14.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.14.0
>
>
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedMultiInput&env=2
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedTwoInput&env=2
> {noformat}
> pnowojski@piotr-mbp: [~/flink - ((no branch, bisect started on pr/16589))] $
> git ls f4afbf3e7de..eb8100f7afe
> eb8100f7afe [4 weeks ago] (pn/bad, bad, refs/bisect/bad)
> [FLINK-22017][coordination] Allow BLOCKING result partition to be
> individually consumable [Thesharing]
> d2005268b1e [4 weeks ago] (HEAD, pn/bisect-4, bisect-4)
> [FLINK-22017][coordination] Get the ConsumedPartitionGroup that
> IntermediateResultPartition and DefaultResultPartition belong to [Thesharing]
> d8b1a6fd368 [3 weeks ago] [FLINK-23372][streaming-java] Disable
> AllVerticesInSameSlotSharingGroupByDefault in batch mode [Timo Walther]
> 4a78097d038 [3 weeks ago] (pn/bisect-3, bisect-3,
> refs/bisect/good-4a78097d0385749daceafd8326930c8cc5f26f1a)
> [FLINK-21928][clients][runtime] Introduce static method constructors of
> DuplicateJobSubmissionException for better readability. [David Moravek]
> 172b9e32215 [3 weeks ago] [FLINK-21928][clients] JobManager failover should
> succeed, when trying to resubmit already terminated job in application mode.
> [David Moravek]
> f483008db86 [3 weeks ago] [FLINK-21928][core] Introduce
> org.apache.flink.util.concurrent.FutureUtils#handleException method, that
> allows future to recover from the specied exception. [David Moravek]
> d7ac08c2ac0 [3 weeks ago] (pn/bisect-2, bisect-2,
> refs/bisect/good-d7ac08c2ac06b9ff31707f3b8f43c07817814d4f)
> [FLINK-22843][docs-zh] Document and code are inconsistent [ZhiJie Yang]
> 16c3ea427df [3 weeks ago] [hotfix] Split the final checkpoint related tests
> to a separate test class. [Yun Gao]
> 31b3d37a22c [7 weeks ago] [FLINK-21089][runtime] Skip the execution of new
> sources if finished on restore [Yun Gao]
> 20fe062e1b5 [3 weeks ago] [FLINK-21089][runtime] Skip execution for the
> legacy source task if finished on restore [Yun Gao]
> 874c627114b [3 weeks ago] [FLINK-21089][runtime] Skip the lifecycle method of
> operators if finished on restore [Yun Gao]
> ceaf24b1d88 [3 weeks ago] (pn/bisect-1, bisect-1,
> refs/bisect/good-ceaf24b1d881c2345a43f305d40435519a09cec9) [hotfix] Fix
> isClosed() for operator wrapper and proxy operator close to the operator
> chain [Yun Gao]
> 41ea591a6db [3 weeks ago] [FLINK-22627][runtime] Remove unused slot request
> protocol [Yangze Guo]
> 489346b60f8 [3 months ago] [FLINK-22627][runtime] Remove PendingSlotRequest
> [Yangze Guo]
> 8ffb4d2af36 [3 months ago] [FLINK-22627][runtime] Remove TaskManagerSlot
> [Yangze Guo]
> 72073741588 [3 months ago] [FLINK-22627][runtime] Remove SlotManagerImpl and
> its related tests [Yangze Guo]
> bdb3b7541b3 [3 months ago] [hotfix][yarn] Remove unused internal options in
> YarnConfigOptionsInternal [Yangze Guo]
> a6a9b192eac [3 weeks ago] [FLINK-23201][streaming] Reset alignment only for
> the currently processed checkpoint [Anton Kalashnikov]
> b35701a35c7 [3 weeks ago] [FLINK-23201][streaming] Calculate checkpoint
> alignment time only for last started checkpoint [Anton Kalashnikov]
> 3abec22c536 [3 weeks ago] [FLINK-23107][table-runtime] Separate
> implementation of deduplicate rank from other rank functions [Shuo Cheng]
> 1a195f5cc59 [3 weeks ago] [FLINK-16093][docs-zh] Translate "System Functions"
> page of "Functions" into Chinese (#16348) [ZhiJie Yang]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-23560) Performance regression on 29.07.2021
[ https://issues.apache.org/jira/browse/FLINK-23560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reassigned FLINK-23560: -- Assignee: Anton Kalashnikov (was: Piotr Nowojski) > Performance regression on 29.07.2021 > > > Key: FLINK-23560 > URL: https://issues.apache.org/jira/browse/FLINK-23560 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.14.0 >Reporter: Piotr Nowojski >Assignee: Anton Kalashnikov >Priority: Blocker > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: Screenshot 2021-07-30 at 15.46.54.png > > > http://codespeed.dak8s.net:8000/timeline/?ben=remoteFilePartition&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=uncompressedMmapPartition&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=compressedFilePartition&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=tupleKeyBy&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=arrayKeyBy&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=uncompressedFilePartition&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=sortedTwoInput&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=sortedMultiInput&env=2 > http://codespeed.dak8s.net:8000/timeline/?ben=globalWindow&env=2 > (And potentially other benchmarks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23221) Docker image vulnerability
[ https://issues.apache.org/jira/browse/FLINK-23221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392046#comment-17392046 ] Till Rohrmann commented on FLINK-23221: --- Downgraded to critical because there is no fix available at this point in time. > Docker image vulnerability > -- > > Key: FLINK-23221 > URL: https://issues.apache.org/jira/browse/FLINK-23221 > Project: Flink > Issue Type: Improvement > Components: flink-docker >Affects Versions: 1.13.1 > Environment: Issue was discovered by AWS ECR image scanning on > apache/flink:1.13.1-scala_2.12 >Reporter: Razvan AGAPE >Priority: Critical > Labels: docker, flink, glibc > Fix For: 1.14.0, 1.13.3 > > > The AWS ECR image scanning reports some HIGH vulnerabilities on > apache/flink:1.13.1-scala_2.12 docker image. In addition, all versions prior > to this one have these issues. > The vulnerabilities are the following: > # [CVE-2021-33574|https://security-tracker.debian.org/tracker/CVE-2021-33574] > # [CVE-2019-25013 - for this one a patch was been released in glibc version > 2.31-9|https://security-tracker.debian.org/tracker/CVE-2019-25013] > Our security policy do not allow us to deploy images having security > vulnerabilities. Searching through the Internet I found that for the first > problem, a patch containing the solution will be release this year. > Do you plan to release a new image containing the newer glibc version in > order to solve those issues? > Also, I checked and the alpine based flink images do not have these > vulnerabilities. Do you plan to release newer versions of flink based on > alpine (latest one is flink:1.8.x)? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23221) Docker image vulnerability
[ https://issues.apache.org/jira/browse/FLINK-23221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-23221: -- Priority: Critical (was: Blocker) > Docker image vulnerability > -- > > Key: FLINK-23221 > URL: https://issues.apache.org/jira/browse/FLINK-23221 > Project: Flink > Issue Type: Improvement > Components: flink-docker >Affects Versions: 1.13.1 > Environment: Issue was discovered by AWS ECR image scanning on > apache/flink:1.13.1-scala_2.12 >Reporter: Razvan AGAPE >Priority: Critical > Labels: docker, flink, glibc > Fix For: 1.14.0, 1.13.3 > > > The AWS ECR image scanning reports some HIGH vulnerabilities on > apache/flink:1.13.1-scala_2.12 docker image. In addition, all versions prior > to this one have these issues. > The vulnerabilities are the following: > # [CVE-2021-33574|https://security-tracker.debian.org/tracker/CVE-2021-33574] > # [CVE-2019-25013 - for this one a patch was been released in glibc version > 2.31-9|https://security-tracker.debian.org/tracker/CVE-2019-25013] > Our security policy do not allow us to deploy images having security > vulnerabilities. Searching through the Internet I found that for the first > problem, a patch containing the solution will be release this year. > Do you plan to release a new image containing the newer glibc version in > order to solve those issues? > Also, I checked and the alpine based flink images do not have these > vulnerabilities. Do you plan to release newer versions of flink based on > alpine (latest one is flink:1.8.x)? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21915) Optimize Execution#finishPartitionsAndUpdateConsumers
[
https://issues.apache.org/jira/browse/FLINK-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhilong Hong closed FLINK-21915.
Resolution: Later
Due to FLINK-22017, now blocking partitions will be individually consumable
once it's finished. {{finishPartitionAndUpdateConsumer}} will be called every
partition is finished. The proposed optimization is no longer valid.
Furthermore, this function is called only when there is intra-region edges in
the graph. In this case the downstream vertices are DEPLOYING/RUNNING when the
upstream vertices are FINISHED. The scenario is rare. Thus for now we just
close it. If there's a new idea about it, we'd like to reopen it.
> Optimize Execution#finishPartitionsAndUpdateConsumers
> -
>
> Key: FLINK-21915
> URL: https://issues.apache.org/jira/browse/FLINK-21915
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Zhilong Hong
>Priority: Major
> Labels: auto-unassigned, pull-request-available
>
> Based on the scheduler benchmark {{PartitionReleaseInBatchJobBenchmark}}
> introduced in FLINK-20612, we find that there's another procedure that has
> O(N^2) computation complexity:
> {{Execution#finishPartitionsAndUpdateConsumers}}.
> Once an execution is finished, it will finish all its BLOCKING partitions and
> update the partition info to all consumer vertices. The procedure can be
> illustrated as the following pseudo code:
> {code:java}
> for all Execution in ExecutionGraph:
> for all produced IntermediateResultPartition of the Execution:
> for all consumer ExecutionVertex of the IntermediateResultPartition:
> update or cache partition info{code}
> This procedure has O(N^2) complexity in total.
> Based on FLINK-21326, the consumed partitions are grouped if they are
> connected to the same consumer vertices. Therefore, we can update partition
> info of the entire ConsumedPartitionGroup in batch, rather than one by one.
> This will decrease the complexity from O(N^2) to O(N).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dmvk commented on a change in pull request #16654: [FLINK-23492][runtime] harden testCachedStatsCleanedAfterCleanupInterval
dmvk commented on a change in pull request #16654:
URL: https://github.com/apache/flink/pull/16654#discussion_r681497890
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/webmonitor/threadinfo/JobVertexThreadInfoTrackerTest.java
##
@@ -139,49 +145,69 @@ public void
testCachedStatsNotUpdatedWithinRefreshInterval() throws Exception {
/** Tests that cached result is NOT reused after refresh interval. */
@Test
public void testCachedStatsUpdatedAfterRefreshInterval() throws Exception {
-final Duration threadInfoStatsRefreshInterval2 = Duration.ofMillis(10);
-final long waitingTime = threadInfoStatsRefreshInterval2.toMillis() +
10;
+final Duration threadInfoStatsRefreshInterval2 = Duration.ofMillis(1);
-final int requestId2 = 1;
-final JobVertexThreadInfoStats threadInfoStats2 =
+// first entry is in the past, so refresh is triggered immediately
upon fetching it
+final JobVertexThreadInfoStats threadInfoStats =
createThreadInfoStats(
-requestId2, TIME_GAP,
Collections.singletonList(threadInfoSample));
+Instant.now().minus(10, ChronoUnit.SECONDS),
+REQUEST_ID,
+Duration.ofMillis(5),
+Collections.singletonList(threadInfoSample));
+final JobVertexThreadInfoStats threadInfoStats2 =
+createThreadInfoStats(1, TIME_GAP,
Collections.singletonList(threadInfoSample));
+// register a CountDownLatch with the cache so we can await refresh of
the entry
+CountDownLatch latch = new CountDownLatch(1);
+Cache
vertexStatsCache =
+createCache(CLEAN_UP_INTERVAL, new
LatchRemovalListener<>(latch));
final JobVertexThreadInfoTracker tracker =
createThreadInfoTracker(
CLEAN_UP_INTERVAL,
threadInfoStatsRefreshInterval2,
-threadInfoStatsDefaultSample,
+vertexStatsCache,
+threadInfoStats,
threadInfoStats2);
-doInitialRequestAndVerifyResult(tracker);
-// ensure that the previous request "expires"
-Thread.sleep(waitingTime);
+// no stats yet, but the request triggers async collection of stats
+assertFalse(tracker.getVertexStats(JOB_ID,
EXECUTION_JOB_VERTEX).isPresent());
+// block until the async call completes and the first result is
available
+tracker.getResultAvailableFuture().get();
+
+// retrieve the entry, triggering the refresh as side effect
+assertExpectedEqualsReceived(
+threadInfoStats, tracker.getVertexStats(JOB_ID,
EXECUTION_JOB_VERTEX));
+
+// wait until the entry is refreshed, with generous buffer
+assertTrue(latch.await(500, TimeUnit.MILLISECONDS));
+// verify that we get the second result on the next request
Optional result =
tracker.getVertexStats(JOB_ID, EXECUTION_JOB_VERTEX);
-
assertExpectedEqualsReceived(threadInfoStats2, result);
-
-assertNotSame(result.get(), threadInfoStatsDefaultSample);
}
/** Tests that cached results are removed within the cleanup interval. */
@Test
public void testCachedStatsCleanedAfterCleanupInterval() throws Exception {
-final Duration cleanUpInterval2 = Duration.ofMillis(10);
-final long waitingTime = cleanUpInterval2.toMillis() + 10;
+final Duration cleanUpInterval2 = Duration.ofMillis(1);
+// register a CountDownLatch with the cache so we can await expiry of
the entry
+CountDownLatch latch = new CountDownLatch(1);
Review comment:
nit: maybe this could have a more descriptive name, eg. `cacheExpired`
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/webmonitor/threadinfo/JobVertexThreadInfoTrackerTest.java
##
@@ -139,49 +145,69 @@ public void
testCachedStatsNotUpdatedWithinRefreshInterval() throws Exception {
/** Tests that cached result is NOT reused after refresh interval. */
@Test
public void testCachedStatsUpdatedAfterRefreshInterval() throws Exception {
-final Duration threadInfoStatsRefreshInterval2 = Duration.ofMillis(10);
-final long waitingTime = threadInfoStatsRefreshInterval2.toMillis() +
10;
+final Duration threadInfoStatsRefreshInterval2 = Duration.ofMillis(1);
Review comment:
nit: there are many variables with unnecessary suffix (`2`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-23575) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails with AsynchronousException
[
https://issues.apache.org/jira/browse/FLINK-23575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392048#comment-17392048
]
Piotr Nowojski commented on FLINK-23575:
This might be a duplicate of FLINK-23558
CC [~akalashnikov]
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails with
> AsynchronousException
> --
>
> Key: FLINK-23575
> URL: https://issues.apache.org/jira/browse/FLINK-23575
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21246&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=4938
> {code}
> Jul 31 21:32:59 [ERROR] Tests run: 72, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 655.588 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase
> Jul 31 21:32:59 [ERROR] shouldRescaleUnalignedCheckpoint[upscale pipeline
> from 20 to 21, buffersPerChannel =
> 1](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 21.288 s <<< ERROR!
> Jul 31 21:32:59 org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> Jul 31 21:32:59 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> Jul 31 21:32:59 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:159)
> Jul 31 21:32:59 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:543)
> Jul 31 21:32:59 at
> sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
> Jul 31 21:32:59 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 31 21:32:59 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 31 21:32:59 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 31 21:32:59 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 31 21:32:59 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 31 21:32:59 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 31 21:32:59 at org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> Jul 31 21:32:59 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Jul 31 21:32:59 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jul 31 21:32:59 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 31 21:32:59 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 31 21:32:59 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 31 21:32:59 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 31 21:32:59 at org.junit.runners.Suite.runChild(Suite.java:128)
> Jul 31 21:32:59 at org.junit.runners.Suite.runChild(Suite.java:27)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 31 21:32:59 at
> org.junit.runners.P
[jira] [Assigned] (FLINK-23351) FileReadingWatermarkITCase.testWatermarkEmissionWithChaining fails due to "too few watermarks emitted" on azure
[
https://issues.apache.org/jira/browse/FLINK-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-23351:
---
Assignee: Arvid Heise
> FileReadingWatermarkITCase.testWatermarkEmissionWithChaining fails due to
> "too few watermarks emitted" on azure
> ---
>
> Key: FLINK-23351
> URL: https://issues.apache.org/jira/browse/FLINK-23351
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20267&view=logs&j=a549b384-c55a-52c0-c451-00e0477ab6db&t=81f2da51-a161-54c7-5b84-6001fed26530&l=10500
> {code}
> Jul 10 22:53:45 [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 21.065 s <<< FAILURE! - in
> org.apache.flink.test.streaming.api.FileReadingWatermarkITCase
> Jul 10 22:53:45 [ERROR]
> testWatermarkEmissionWithChaining(org.apache.flink.test.streaming.api.FileReadingWatermarkITCase)
> Time elapsed: 20.25 s <<< FAILURE!
> Jul 10 22:53:45 java.lang.AssertionError: too few watermarks emitted in 3057
> ms expected:<305.0> but was:<124.0>
> Jul 10 22:53:45 at org.junit.Assert.fail(Assert.java:89)
> Jul 10 22:53:45 at org.junit.Assert.failNotEquals(Assert.java:835)
> Jul 10 22:53:45 at org.junit.Assert.assertEquals(Assert.java:555)
> Jul 10 22:53:45 at
> org.apache.flink.test.streaming.api.FileReadingWatermarkITCase.testWatermarkEmissionWithChaining(FileReadingWatermarkITCase.java:79)
> Jul 10 22:53:45 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 10 22:53:45 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 10 22:53:45 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 10 22:53:45 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 10 22:53:45 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 10 22:53:45 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 10 22:53:45 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 10 22:53:45 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 10 22:53:45 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 10 22:53:45 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 10 22:53:45 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 10 22:53:45 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 10 22:53:45 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> Jul 10 22:53:45 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jir
[GitHub] [flink] JingsongLi commented on a change in pull request #16630: [FLINK-23531][table]Allow skip all change log for row-time deduplicate mini-batch
JingsongLi commented on a change in pull request #16630:
URL: https://github.com/apache/flink/pull/16630#discussion_r681504149
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecDeduplicate.java
##
@@ -89,6 +91,17 @@
+ "but there will be additional overhead."
+ "Default is true.");
+@Experimental
+public static final ConfigOption
TABLE_EXEC_DEDUPLICATE_MINIBATCH_COMPACT_CHANGES =
+
ConfigOptions.key("table.exec.deduplicate.mini-batch.compact-changes")
+.booleanType()
+.defaultValue(false)
+.withDescription(
+"Set whether send all change log for row-time
mini-batch."
Review comment:
Set whether to compact the changes sent downstream in row-time
mini-batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16679: [FLINK-23402][streaming-java] Simplify shuffle mode for batch execution
zhuzhurk commented on a change in pull request #16679:
URL: https://github.com/apache/flink/pull/16679#discussion_r681487662
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/ExecutionOptions.java
##
@@ -40,10 +40,10 @@
"Runtime execution mode of DataStream programs.
Among other things, "
+ "this controls task scheduling, network
shuffle behavior, and time semantics.");
-public static final ConfigOption SHUFFLE_MODE =
-ConfigOptions.key("execution.shuffle-mode")
-.enumType(ShuffleMode.class)
-.defaultValue(ShuffleMode.AUTOMATIC)
+public static final ConfigOption BATCH_SHUFFLE_MODE =
+ConfigOptions.key("execution.batch-shuffle-mode")
+.enumType(BatchShuffleMode.class)
+.defaultValue(BatchShuffleMode.ALL_EXCHANGES_BLOCKING)
.withDescription(
Review comment:
The description needs to be updated. It does not need to talk about how
it works for streaming mode but can simply say that it only works for batch
mode.
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/BatchShuffleMode.java
##
@@ -41,7 +41,7 @@
* to run upstream and downstream tasks simultaneously.
*/
@PublicEvolving
-public enum ShuffleMode implements DescribedEnum {
+public enum BatchShuffleMode implements DescribedEnum {
Review comment:
The class java doc needs to be updated to remove statements of streaming
mode.
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/BatchShuffleMode.java
##
@@ -41,7 +41,7 @@
* to run upstream and downstream tasks simultaneously.
*/
@PublicEvolving
-public enum ShuffleMode implements DescribedEnum {
+public enum BatchShuffleMode implements DescribedEnum {
/**
* Upstream and downstream tasks run simultaneously.
Review comment:
> This is the only supported shuffle behavior in streaming mode.
This statement should be removed from both the java doc and the config
description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #16630: [FLINK-23531][table]Allow skip all change log for row-time deduplicate mini-batch
JingsongLi commented on a change in pull request #16630:
URL: https://github.com/apache/flink/pull/16630#discussion_r681506823
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecDeduplicate.java
##
@@ -89,6 +91,17 @@
+ "but there will be additional overhead."
+ "Default is true.");
+@Experimental
+public static final ConfigOption
TABLE_EXEC_DEDUPLICATE_MINIBATCH_COMPACT_CHANGES =
+
ConfigOptions.key("table.exec.deduplicate.mini-batch.compact-changes")
+.booleanType()
+.defaultValue(false)
+.withDescription(
+"Set whether send all change log for row-time
mini-batch."
++ "If true, Flink will only send the
latest change log to downstream like proc-time."
Review comment:
If true, Flink will compact changes, only send the latest change to
downstream. Notes: If the downstream needs the details of versioned data, this
optimization cannot be opened.
If false, Flink will send all changes to downstream just like when the
mini-batch is not on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16465: [FLINK-22910][runtime] Refine ShuffleMaster lifecycle management for pluggable shuffle service framework
flinkbot edited a comment on pull request #16465: URL: https://github.com/apache/flink/pull/16465#issuecomment-878076127 ## CI report: * 16a88a240fcebac5844190058e1645b5d1e26e2a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21139) * 6608bdbcb23d05495d6e232fe57e49560edfaa05 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21359) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16589: [FLINK-23541] Call StreamOperator#finish on EndOfData
flinkbot edited a comment on pull request #16589: URL: https://github.com/apache/flink/pull/16589#issuecomment-886429447 ## CI report: * 791a3f1825b2e2d5682c40e9b7e52a4cbc576429 UNKNOWN * a29a6115a578c543a9856a07d0ef58f79ebcc895 UNKNOWN * c679fefa6bc59735ea1fb7f3ed6a5cd74685d288 UNKNOWN * cd0181abe1514ca324996415bca37669d879d350 UNKNOWN * 1453bb994c02b90aab31e8c5d1f4ac6ae1452e06 UNKNOWN * e6574235f235d7377181c96e0390beac5d8a7ae0 UNKNOWN * cbf860c54288c7da244fd439353bf275440b00c7 UNKNOWN * bf8281f701b162aa87ce2ae6245d08fe62633fe9 UNKNOWN * c4067989d48b67b6d2f3974d924d3c8e96df9684 UNKNOWN * 0f87c952fb7a0d99c09d1ef193f75bcf110e5f2a UNKNOWN * c2ac0201ddeb1643fdfb85e0872ddacf64c6fc37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21329) * f85d7f9894d68c82f1e01aa317d961e5d686182a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16626: [FLINK-23498][table] Introduce a full PlannerConfiguration
flinkbot edited a comment on pull request #16626: URL: https://github.com/apache/flink/pull/16626#issuecomment-888274470 ## CI report: * 1faf4603d913d2403044bc38bf57e07d3fd20a1e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21117) * b7a07005f53c491bbd2eb9437c479ef31581bf2a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16633: [FLINK-21080][runtime][checkpoint] Handle UnionListState with finished operators
flinkbot edited a comment on pull request #16633: URL: https://github.com/apache/flink/pull/16633#issuecomment-888924880 ## CI report: * aedb0fe1f96c4a7acdf5b5e49d45008354b3963e UNKNOWN * 6ffe91205a7123699f66781f14b9f3a69347fdec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21301) * 329c1645c5e808c80db672f95e3f3a4646b1fd4a UNKNOWN * c6650fe4b9749b053830a8a2220e6d5ba7fa02e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21362) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16630: [FLINK-23531][table]Allow skip all change log for row-time deduplicate mini-batch
flinkbot edited a comment on pull request #16630: URL: https://github.com/apache/flink/pull/16630#issuecomment-17597 ## CI report: * feb601f7603dad7defb60305c1ea603ac2fc1b24 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21141) * dd3f5c216993d99c6742add9f4e878a54b52343a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16640: [FLINK-22891][runtime] Using CompletableFuture to sync the scheduling…
flinkbot edited a comment on pull request #16640: URL: https://github.com/apache/flink/pull/16640#issuecomment-889598196 ## CI report: * ed73065a1c804251ca5731fa860dadd4fcf7d16d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21348) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16638: [FLINK-23513] Remove legacy descriptors
flinkbot edited a comment on pull request #16638: URL: https://github.com/apache/flink/pull/16638#issuecomment-889186834 ## CI report: * 7dfa8736f6e9a4f97799dce51a318965d2e16344 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21305) * a08a51997cb8a8b63ef73c096967e7060d35ba94 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16674: [FLINK-23587][deployment] Set Deployment's Annotation when using kubernetes
flinkbot edited a comment on pull request #16674: URL: https://github.com/apache/flink/pull/16674#issuecomment-890963322 ## CI report: * 97c698f8c91132a435be2eab8c39c903d6d014bc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21349) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16679: [FLINK-23402][streaming-java] Simplify shuffle mode for batch execution
flinkbot edited a comment on pull request #16679: URL: https://github.com/apache/flink/pull/16679#issuecomment-891160011 ## CI report: * 6138c21876402893fccefeff887d756665180c3a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21318) * 439128217c8d5f8aa0ad971075a6956b1103d190 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21363) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Tartarus0zm commented on pull request #16630: [FLINK-23531][table]Allow skip all change log for row-time deduplicate mini-batch
Tartarus0zm commented on pull request #16630: URL: https://github.com/apache/flink/pull/16630#issuecomment-891613311 @JingsongLi thanks for your review. I fixed it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] NicoK merged pull request #16671: [BP-1.13][FLINK-23102][rest] Return empty FlameGraph if feature is disabled
NicoK merged pull request #16671: URL: https://github.com/apache/flink/pull/16671 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23102) Accessing FlameGraphs while not being enabled returns an exception
[ https://issues.apache.org/jira/browse/FLINK-23102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392069#comment-17392069 ] Nico Kruber commented on FLINK-23102: - release-1.13: 39ad632a60896c94fb796e46c1bacfea32ea68d8 > Accessing FlameGraphs while not being enabled returns an exception > -- > > Key: FLINK-23102 > URL: https://issues.apache.org/jira/browse/FLINK-23102 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.13.1 >Reporter: Nico Kruber >Assignee: Yao Zhang >Priority: Major > Labels: pull-request-available, usability > Fix For: 1.14.0 > > Attachments: image-2021-06-22-17-36-47-730.png > > > Trying to retrieve the FlameGraph in a job that doesn't have it enabled > returns this ugly exception: > !image-2021-06-22-17-36-47-730.png! > Instead, it could mention that this feature is not enabled and describe how > to enable it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23102) Accessing FlameGraphs while not being enabled returns an exception
[ https://issues.apache.org/jira/browse/FLINK-23102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber updated FLINK-23102: Fix Version/s: 1.13.3 > Accessing FlameGraphs while not being enabled returns an exception > -- > > Key: FLINK-23102 > URL: https://issues.apache.org/jira/browse/FLINK-23102 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.13.1 >Reporter: Nico Kruber >Assignee: Yao Zhang >Priority: Major > Labels: pull-request-available, usability > Fix For: 1.14.0, 1.13.3 > > Attachments: image-2021-06-22-17-36-47-730.png > > > Trying to retrieve the FlameGraph in a job that doesn't have it enabled > returns this ugly exception: > !image-2021-06-22-17-36-47-730.png! > Instead, it could mention that this feature is not enabled and describe how > to enable it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] NicoK closed pull request #16419: [FLINK-23101][runtime] Flame Graphs initial view says it is 18800 day…
NicoK closed pull request #16419: URL: https://github.com/apache/flink/pull/16419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] NicoK commented on pull request #16419: [FLINK-23101][runtime] Flame Graphs initial view says it is 18800 day…
NicoK commented on pull request #16419: URL: https://github.com/apache/flink/pull/16419#issuecomment-891617120 Fixed via #16352. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-22527) Give friendly hint when click FlameGraph without rest.flamegraph.enabled
[
https://issues.apache.org/jira/browse/FLINK-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber resolved FLINK-22527.
-
Fix Version/s: (was: 1.13.3)
(was: 1.14.0)
Resolution: Duplicate
> Give friendly hint when click FlameGraph without rest.flamegraph.enabled
>
>
> Key: FLINK-22527
> URL: https://issues.apache.org/jira/browse/FLINK-22527
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Yun Tang
>Priority: Minor
> Labels: auto-deprioritized-major
>
> Currently, if {{rest.flamegraph.enabled}} is not enabled, the webUI would
> just tell user that "Unable to load requested file ", however, it should
> give more friendly hint that to enable {{rest.flamegraph.enabled}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-23101) Flame Graphs initial view says it is 18800 days in the past
[ https://issues.apache.org/jira/browse/FLINK-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber resolved FLINK-23101. - Fix Version/s: 1.13.3 1.14.0 Resolution: Fixed > Flame Graphs initial view says it is 18800 days in the past > --- > > Key: FLINK-23101 > URL: https://issues.apache.org/jira/browse/FLINK-23101 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.13.1 >Reporter: Nico Kruber >Assignee: Yao Zhang >Priority: Minor > Labels: pull-request-available, usability > Fix For: 1.14.0, 1.13.3 > > Attachments: image.png > > > When you look at the Flame Graphs for a task for the first time, it will show > an empty space and say that the measurement was ~18800 days in the past (see > the attached image). > > This should rather be something more useful like "no measurement yet" or so... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23595) Allow JSON format deserialize non-numeric numbers
loyi created FLINK-23595:
Summary: Allow JSON format deserialize non-numeric numbers
Key: FLINK-23595
URL: https://issues.apache.org/jira/browse/FLINK-23595
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.13.1
Environment: {noformat}
Exception stack:
Caused by: java.io.IOException: Failed to deserialize JSON
'{"uniqueId":"6974697215254697525","monitorKey":"live-log-list","indicatorMap":{"record-retry-rate":NaN,"request-latency-max":-Infinity,"request-latency-avg":NaN,"buffer-available-bytes":3.3554432E7,"waiting-threads":0.0,"record-error-rate":NaN},"tagMap":{"_aggregate":"RAW","host":"live-log-list-001"},"periodInMs":0,"dataTime":1627903774962,"receiveTime":1627903774965}'.
at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:149)
~[?:?]
at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:81)
~[?:?]
at
org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializationSchemaWrapper.deserialize(KafkaDeserializationSchemaWrapper.java:58)
~[?:?]
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
~[?:?]
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
~[?:?]
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
~[?:?]
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
Caused by:
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Non-standard token 'NaN': enable JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS
to allow
at [Source: UNKNOWN; line: 1, column: 310]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2337)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:710)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2669)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1094)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:268)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:277)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:69)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:16)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4635)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3056)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:142)
~[?:?]
at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:81)
~[?:?]
at
org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
~[flink-dist_2.11-1.13.1.jar:1.13.1]
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializ
[jira] [Created] (FLINK-23596) flink on k8s can create more than one instance
zhouwenyang created FLINK-23596: --- Summary: flink on k8s can create more than one instance Key: FLINK-23596 URL: https://issues.apache.org/jira/browse/FLINK-23596 Project: Flink Issue Type: New Feature Reporter: zhouwenyang Like spark use --conf spark.executor.instances=2，I hope flink can support similar paramters -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23597) support Add Jar in Table api
zoucao created FLINK-23597: -- Summary: support Add Jar in Table api Key: FLINK-23597 URL: https://issues.apache.org/jira/browse/FLINK-23597 Project: Flink Issue Type: Improvement Reporter: zoucao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] curcur commented on pull request #16653: [FLINK-23558][streaming] Ignoring RejectedExecutionException during s…
curcur commented on pull request #16653: URL: https://github.com/apache/flink/pull/16653#issuecomment-891622506 I have a bit concern of this: this change swallows all "RejectedExecutionException" no matter it is because of task finish or not. I think the right fix is not to submit mailbox execution after the task finishes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #16653: [FLINK-23558][streaming] Ignoring RejectedExecutionException during s…
curcur edited a comment on pull request #16653: URL: https://github.com/apache/flink/pull/16653#issuecomment-891622506 I have a bit concern of this fix: this change swallows all "RejectedExecutionException" no matter it is because of task finish or not. I think the right fix is not to submit mailbox execution after the task finishes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23597) support 'add jar' in table api
[ https://issues.apache.org/jira/browse/FLINK-23597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoucao updated FLINK-23597: --- Summary: support 'add jar' in table api (was: support Add Jar in Table api) > support 'add jar' in table api > -- > > Key: FLINK-23597 > URL: https://issues.apache.org/jira/browse/FLINK-23597 > Project: Flink > Issue Type: Improvement >Reporter: zoucao >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] curcur edited a comment on pull request #16653: [FLINK-23558][streaming] Ignoring RejectedExecutionException during s…
curcur edited a comment on pull request #16653: URL: https://github.com/apache/flink/pull/16653#issuecomment-891622506 I have a bit concern of this fix: this change swallows all "RejectedExecutionException" no matter it is because of task finish or not. I think the right fix is not to submit mailbox execution after the task finishes, or at least identify this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15304: [FLINK-20731][connector] Introduce Pulsar Source
flinkbot edited a comment on pull request #15304: URL: https://github.com/apache/flink/pull/15304#issuecomment-803430086 ## CI report: * 04da1862ec1f107c74017283c916229b560d9731 UNKNOWN * 4dcee9f07135401160aae3f0d01bd480630f808f UNKNOWN * d326abc93378bb3c4b53616e7717c409d9876ade UNKNOWN * e87a3a4c253ba99ebb42333f54b5ffb0f024e6c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21339) * 2af90ed68fc1cf845251e606ced17c9017b8f165 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #16653: [FLINK-23558][streaming] Ignoring RejectedExecutionException during s…
curcur edited a comment on pull request #16653: URL: https://github.com/apache/flink/pull/16653#issuecomment-891622506 I have a bit concern of this fix: this change swallows all "RejectedExecutionException" no matter it is because of task finish or not. I think the right way is not to submit mailbox execution after the task finishes, or at least identify this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shutter-cp opened a new pull request #16684: FIX FLINK-23580
shutter-cp opened a new pull request #16684: URL: https://github.com/apache/flink/pull/16684 add clinkHouse JDBC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #16653: [FLINK-23558][streaming] Ignoring RejectedExecutionException during s…
curcur edited a comment on pull request #16653: URL: https://github.com/apache/flink/pull/16653#issuecomment-891622506 I have a bit concern of this fix: this change swallows all "RejectedExecutionException" no matter it is because of task finish or not. I think the right way is not to submit mailbox execution after the task finishes, or at least identify this case. I think there should be similar problems for other registered timers, maybe we can also unregister this timer before task finish. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #16653: [FLINK-23558][streaming] Ignoring RejectedExecutionException during s…
curcur edited a comment on pull request #16653: URL: https://github.com/apache/flink/pull/16653#issuecomment-891622506 I have a bit concern of this fix: this change swallows all "RejectedExecutionException" no matter it is because of task finish or not. The right way is not to submit mailbox execution after the task finishes, or at least identify this case. I think there should be similar problems for other registered timers, maybe we can also unregister this timer before the task finishes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23597) support 'add jar' in table api
[ https://issues.apache.org/jira/browse/FLINK-23597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoucao updated FLINK-23597: --- Component/s: Table SQL / API Description: Now, sql-client has supported 'add jar' semantic， I think we should support it in 'TableEnvironment'. > support 'add jar' in table api > -- > > Key: FLINK-23597 > URL: https://issues.apache.org/jira/browse/FLINK-23597 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: zoucao >Priority: Major > > Now, sql-client has supported 'add jar' semantic， I think we should support > it in 'TableEnvironment'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23580) Cannot handle such jdbc url
[
https://issues.apache.org/jira/browse/FLINK-23580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-23580:
---
Labels: pull-request-available (was: )
> Cannot handle such jdbc url
> ---
>
> Key: FLINK-23580
> URL: https://issues.apache.org/jira/browse/FLINK-23580
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.12.0
>Reporter: chenpeng
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-08-02-16-02-21-897.png
>
>
>
> Caused by: java.lang.IllegalStateException: Cannot handle such jdbc url:
> jdbc:clickhouse://xx:8123/dict
> {code:java}
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Failed
> to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to
> no-operation (NOP) logger implementationSLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> org.apache.flink.table.api.ValidationException: Unable to create a source for
> reading table 'default_catalog.default_database.sink_table'.
> Table options are:
> 'connector'='jdbc''driver'='ru.yandex.clickhouse.ClickHouseDriver''password'='''table-name'='tbl3_dict''url'='jdbc:clickhouse://xxx:8123/dict''username'='default'
> at
> org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:125)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:265)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:100)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:165)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:157)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:823)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:795)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:250)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:639)
> at FlinkStreamSql.test7(FlinkStreamSql.java:212) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at
> org.junit.runners.ParentRunner$3.evaluate(Pare
[GitHub] [flink] flinkbot edited a comment on pull request #16368: ignore
flinkbot edited a comment on pull request #16368: URL: https://github.com/apache/flink/pull/16368#issuecomment-873854539 ## CI report: * c8f4d8038e345ae3083ce5c32d5f54ecd812792b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21350) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16576: [FLINK-23463][doc] Replace the tag with ShortCodes in document
flinkbot edited a comment on pull request #16576: URL: https://github.com/apache/flink/pull/16576#issuecomment-885383955 ## CI report: * 3ec79c99b0f88f08f877037e72e7575db41080dc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21351) * e953be8b85c3dfc68e3965cff8a819189cd20eee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16589: [FLINK-23541] Call StreamOperator#finish on EndOfData
flinkbot edited a comment on pull request #16589: URL: https://github.com/apache/flink/pull/16589#issuecomment-886429447 ## CI report: * 791a3f1825b2e2d5682c40e9b7e52a4cbc576429 UNKNOWN * a29a6115a578c543a9856a07d0ef58f79ebcc895 UNKNOWN * c679fefa6bc59735ea1fb7f3ed6a5cd74685d288 UNKNOWN * cd0181abe1514ca324996415bca37669d879d350 UNKNOWN * 1453bb994c02b90aab31e8c5d1f4ac6ae1452e06 UNKNOWN * e6574235f235d7377181c96e0390beac5d8a7ae0 UNKNOWN * cbf860c54288c7da244fd439353bf275440b00c7 UNKNOWN * bf8281f701b162aa87ce2ae6245d08fe62633fe9 UNKNOWN * c4067989d48b67b6d2f3974d924d3c8e96df9684 UNKNOWN * 0f87c952fb7a0d99c09d1ef193f75bcf110e5f2a UNKNOWN * c2ac0201ddeb1643fdfb85e0872ddacf64c6fc37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21329) * f85d7f9894d68c82f1e01aa317d961e5d686182a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21360) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16628: [FLINK-23453][streaming] Created the buffer size calculator for the ability to automatically change the buffer size based on the
akalash commented on a change in pull request #16628:
URL: https://github.com/apache/flink/pull/16628#discussion_r681527264
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
##
@@ -543,35 +543,34 @@
/** The total time for which automated adjusted buffers should be fully
consumed. */
@Documentation.Section(Documentation.Sections.ALL_TASK_MANAGER_NETWORK)
-public static final ConfigOption
AUTOMATIC_BUFFER_ADJUSTMENT_CONSUMPTION_TIME =
-ConfigOptions.key(
-
"taskmanager.network.memory.automatic-buffer-adjustment.consumption-time")
-.intType()
-.defaultValue(1000)
+public static final ConfigOption BUFFER_DEBLOAT_TARGET =
+
ConfigOptions.key("taskmanager.network.memory.buffer-debloat.target")
+.durationType()
+.defaultValue(Duration.ofSeconds(1))
.withDescription(
-"The total time for which automated adjusted
buffers should be fully consumed. "
-+ "This means that the in-flight data
between two subtask should be fully consumed for approximately this time.");
+"The target total time after which buffered
in-flight data should be fully consumed. "
++ "This configuration option will be used,
in combination with the measured throughput, to adjust the amount of in-flight
data.");
@Documentation.Section(Documentation.Sections.ALL_TASK_MANAGER_NETWORK)
-public static final ConfigOption
AUTOMATIC_BUFFER_ADJUSTMENT_ENABLED =
-
ConfigOptions.key("taskmanager.network.memory.automatic-buffer-adjustment.enabled")
+public static final ConfigOption BUFFER_DEBLOAT_ENABLED =
+
ConfigOptions.key("taskmanager.network.memory.buffer-debloat.enabled")
.booleanType()
-.defaultValue(false)
+.defaultValue(true)
Review comment:
Yes, I believe so. Perhaps, I should add my properties to this file too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16640: [FLINK-22891][runtime] Using CompletableFuture to sync the scheduling…
flinkbot edited a comment on pull request #16640: URL: https://github.com/apache/flink/pull/16640#issuecomment-889598196 ## CI report: * ed73065a1c804251ca5731fa860dadd4fcf7d16d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21348) * 81126114e90f8947dba92e4bc1bf785f6fc86ac5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22405) Support fixed-lengh chars in the LeadLag built-in function
[
https://issues.apache.org/jira/browse/FLINK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392072#comment-17392072
]
Nico Kruber commented on FLINK-22405:
-
I think, it is...
(I see no need of maintaining a specialized implementation)
Do you want to work on this, create a test and open a PR? In that case, I would
assign you to this ticket.
> Support fixed-lengh chars in the LeadLag built-in function
> --
>
> Key: FLINK-22405
> URL: https://issues.apache.org/jira/browse/FLINK-22405
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Nico Kruber
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available, starter
>
> LeadLag aggregate function does not support type: ''CHAR'', as in the
> following example (a CAST to VARCHAR works around this). Technically, there
> should be no reason though to support STRING/VARCHAR but not CHAR:
> {code:sql}
> CREATE TEMPORARY VIEW test_cardinality AS
> SELECT * FROM ( VALUES
> ('Alice', 'al...@test.com', ARRAY [ 'al...@test.com' ], 'Test Ltd'),
> ('Alice', 'al...@test.com', ARRAY [ 'al...@test.com' ], 'Test Ltd'),
> ('Alice', 'al...@test2.com', ARRAY [ 'al...@test.com', 'al...@test2.com' ],
> 'Test Ltd'))
> AS t ( name, email, aliases, company );
> {code}
> {code:sql}
> SELECT
> name,
> LEAD(company, 0) AS company
> FROM test_cardinality
> WHERE CARDINALITY(aliases) >= 2
> GROUP BY name;
> {code}
> -> see
> https://github.com/apache/flink/blob/release-1.13.0-rc1/flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/AggFunctionFactory.scala#L331
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16626: [FLINK-23498][table] Introduce a full PlannerConfiguration
flinkbot edited a comment on pull request #16626: URL: https://github.com/apache/flink/pull/16626#issuecomment-888274470 ## CI report: * 1faf4603d913d2403044bc38bf57e07d3fd20a1e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21117) * b7a07005f53c491bbd2eb9437c479ef31581bf2a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21365) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16630: [FLINK-23531][table]Allow skip all change log for row-time deduplicate mini-batch
flinkbot edited a comment on pull request #16630: URL: https://github.com/apache/flink/pull/16630#issuecomment-17597 ## CI report: * feb601f7603dad7defb60305c1ea603ac2fc1b24 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21141) * dd3f5c216993d99c6742add9f4e878a54b52343a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21366) * c588afeb471bcd8c82ffa9149663b1f4b1084b6b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16667: [FLINK-22912][python] Support state ttl in Python DataStream API
flinkbot edited a comment on pull request #16667: URL: https://github.com/apache/flink/pull/16667#issuecomment-890801858 ## CI report: * 207ff3a175da76f1e0878aa0a9788dbe478c83f0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21343) * eca746e23bdf0ebfaeba1c5a14ca50262241dec5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16638: [FLINK-23513] Remove legacy descriptors
flinkbot edited a comment on pull request #16638: URL: https://github.com/apache/flink/pull/16638#issuecomment-889186834 ## CI report: * 7dfa8736f6e9a4f97799dce51a318965d2e16344 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21305) * a08a51997cb8a8b63ef73c096967e7060d35ba94 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21367) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16684: FIX FLINK-23580
flinkbot commented on pull request #16684: URL: https://github.com/apache/flink/pull/16684#issuecomment-891625963 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5493c026a97540ffc65abe158545e0236b4d9e39 (Tue Aug 03 07:59:19 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-23580).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23593) Performance regression on 15.07.2021
[
https://issues.apache.org/jira/browse/FLINK-23593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392073#comment-17392073
]
Timo Walther commented on FLINK-23593:
--
I'm wondering if we should simply ignore this regression. The benchmark is
highly simplified and does not consider complex pipeline regions. The batch
mode of DataStream API is still in early status and the changed configuration
option actually just ensures consistency with the Blink planner. [~zhuzh] what
is your opinion here?
> Performance regression on 15.07.2021
>
>
> Key: FLINK-23593
> URL: https://issues.apache.org/jira/browse/FLINK-23593
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Benchmarks
>Affects Versions: 1.14.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.14.0
>
>
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedMultiInput&env=2
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedTwoInput&env=2
> {noformat}
> pnowojski@piotr-mbp: [~/flink - ((no branch, bisect started on pr/16589))] $
> git ls f4afbf3e7de..eb8100f7afe
> eb8100f7afe [4 weeks ago] (pn/bad, bad, refs/bisect/bad)
> [FLINK-22017][coordination] Allow BLOCKING result partition to be
> individually consumable [Thesharing]
> d2005268b1e [4 weeks ago] (HEAD, pn/bisect-4, bisect-4)
> [FLINK-22017][coordination] Get the ConsumedPartitionGroup that
> IntermediateResultPartition and DefaultResultPartition belong to [Thesharing]
> d8b1a6fd368 [3 weeks ago] [FLINK-23372][streaming-java] Disable
> AllVerticesInSameSlotSharingGroupByDefault in batch mode [Timo Walther]
> 4a78097d038 [3 weeks ago] (pn/bisect-3, bisect-3,
> refs/bisect/good-4a78097d0385749daceafd8326930c8cc5f26f1a)
> [FLINK-21928][clients][runtime] Introduce static method constructors of
> DuplicateJobSubmissionException for better readability. [David Moravek]
> 172b9e32215 [3 weeks ago] [FLINK-21928][clients] JobManager failover should
> succeed, when trying to resubmit already terminated job in application mode.
> [David Moravek]
> f483008db86 [3 weeks ago] [FLINK-21928][core] Introduce
> org.apache.flink.util.concurrent.FutureUtils#handleException method, that
> allows future to recover from the specied exception. [David Moravek]
> d7ac08c2ac0 [3 weeks ago] (pn/bisect-2, bisect-2,
> refs/bisect/good-d7ac08c2ac06b9ff31707f3b8f43c07817814d4f)
> [FLINK-22843][docs-zh] Document and code are inconsistent [ZhiJie Yang]
> 16c3ea427df [3 weeks ago] [hotfix] Split the final checkpoint related tests
> to a separate test class. [Yun Gao]
> 31b3d37a22c [7 weeks ago] [FLINK-21089][runtime] Skip the execution of new
> sources if finished on restore [Yun Gao]
> 20fe062e1b5 [3 weeks ago] [FLINK-21089][runtime] Skip execution for the
> legacy source task if finished on restore [Yun Gao]
> 874c627114b [3 weeks ago] [FLINK-21089][runtime] Skip the lifecycle method of
> operators if finished on restore [Yun Gao]
> ceaf24b1d88 [3 weeks ago] (pn/bisect-1, bisect-1,
> refs/bisect/good-ceaf24b1d881c2345a43f305d40435519a09cec9) [hotfix] Fix
> isClosed() for operator wrapper and proxy operator close to the operator
> chain [Yun Gao]
> 41ea591a6db [3 weeks ago] [FLINK-22627][runtime] Remove unused slot request
> protocol [Yangze Guo]
> 489346b60f8 [3 months ago] [FLINK-22627][runtime] Remove PendingSlotRequest
> [Yangze Guo]
> 8ffb4d2af36 [3 months ago] [FLINK-22627][runtime] Remove TaskManagerSlot
> [Yangze Guo]
> 72073741588 [3 months ago] [FLINK-22627][runtime] Remove SlotManagerImpl and
> its related tests [Yangze Guo]
> bdb3b7541b3 [3 months ago] [hotfix][yarn] Remove unused internal options in
> YarnConfigOptionsInternal [Yangze Guo]
> a6a9b192eac [3 weeks ago] [FLINK-23201][streaming] Reset alignment only for
> the currently processed checkpoint [Anton Kalashnikov]
> b35701a35c7 [3 weeks ago] [FLINK-23201][streaming] Calculate checkpoint
> alignment time only for last started checkpoint [Anton Kalashnikov]
> 3abec22c536 [3 weeks ago] [FLINK-23107][table-runtime] Separate
> implementation of deduplicate rank from other rank functions [Shuo Cheng]
> 1a195f5cc59 [3 weeks ago] [FLINK-16093][docs-zh] Translate "System Functions"
> page of "Functions" into Chinese (#16348) [ZhiJie Yang]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16683: [hotfix][docs]improve PushGatewayReporter config description
flinkbot edited a comment on pull request #16683: URL: https://github.com/apache/flink/pull/16683#issuecomment-891515480 ## CI report: * 3b3f16a9694732d6211a63d324d4bb579933fe87 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21354) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16628: [FLINK-23453][streaming] Created the buffer size calculator for the ability to automatically change the buffer size based on the
akalash commented on a change in pull request #16628:
URL: https://github.com/apache/flink/pull/16628#discussion_r681528714
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/throughput/ThroughputCalculator.java
##
@@ -71,4 +71,8 @@ public long calculateThroughput() {
return lastThroughput;
}
+
+public long lastThroughput() {
Review comment:
good catch. Initially, I refactored in such a way that I would needed
this getter but then I changed my mind but forgot to remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-23450) Properties map is not set in DebeziumAvroFormatFactory
[ https://issues.apache.org/jira/browse/FLINK-23450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-23450. Fix Version/s: 1.13.3 1.12.6 1.14.0 Resolution: Fixed Fixed in 1.14: 78db0e7c50235d7f5ae83225ac11192f5b121f7a Fixed in 1.13: 55981a0ec8576da7776ef2d0fb803d7bcd905036 Fixed in 1.12: 33ad09863a81711ff79b040e12eabb051470ba40 > Properties map is not set in DebeziumAvroFormatFactory > -- > > Key: FLINK-23450 > URL: https://issues.apache.org/jira/browse/FLINK-23450 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.12.6, 1.13.3 > > > FLINK-21229 did not set the properties map correctly in > DebeziumAvroFormatFactory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] twalthr commented on a change in pull request #16679: [FLINK-23402][streaming-java] Simplify shuffle mode for batch execution
twalthr commented on a change in pull request #16679:
URL: https://github.com/apache/flink/pull/16679#discussion_r681532037
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/BatchShuffleMode.java
##
@@ -41,7 +41,7 @@
* to run upstream and downstream tasks simultaneously.
*/
@PublicEvolving
-public enum ShuffleMode implements DescribedEnum {
+public enum BatchShuffleMode implements DescribedEnum {
Review comment:
I thought it might be helpful information for users to properly
understand the difference between streaming mode and batch mode. But I can
remove it if you think it is rather confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-23558) E2e tests fail because of quiesced system timers service
[
https://issues.apache.org/jira/browse/FLINK-23558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392084#comment-17392084
]
Dawid Wysakowicz commented on FLINK-23558:
--
Copied over from the duplicate:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21255&view=logs&j=a549b384-c55a-52c0-c451-00e0477ab6db&t=eef5922c-08d9-5ba3-7299-8393476594e7&l=10654
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21246&view=logs&j=d8d26c26-7ec2-5ed2-772e-7a1a1eb8317c&t=ec8797b0-5eee-5a0e-f936-8db65cff44cc&l=10540
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21246&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=4938
> E2e tests fail because of quiesced system timers service
>
>
> Key: FLINK-23558
> URL: https://issues.apache.org/jira/browse/FLINK-23558
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Assignee: Anton Kalashnikov
>Priority: Blocker
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21180&view=logs&j=739e6eac-8312-5d31-d437-294c4d26fced&t=2a8cc459-df7a-5e6f-12bf-96efcc369aa9&l=10484
> {code}
> Jul 29 21:41:15 Caused by:
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailbox$MailboxClosedException:
> Mailbox is in state QUIESCED, but is required to be in state OPEN for put
> operations.
> Jul 29 21:41:15 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkPutStateConditions(TaskMailboxImpl.java:269)
> ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
> Jul 29 21:41:15 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.put(TaskMailboxImpl.java:197)
> ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
> Jul 29 21:41:15 at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.execute(MailboxExecutorImpl.java:74)
> ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
> Jul 29 21:41:15 at
> org.apache.flink.runtime.mailbox.MailboxExecutor.submit(MailboxExecutor.java:163)
> ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
> Jul 29 21:41:15 at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$throughputCalculationSetup$3(StreamTask.java:688)
> ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
> Jul 29 21:41:15 at
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$ScheduledTask.run(SystemProcessingTimeService.java:317)
> ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
> Jul 29 21:41:15 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_302]
> Jul 29 21:41:15 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_302]
> Jul 29 21:41:15 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> ~[?:1.8.0_302]
> Jul 29 21:41:15 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> ~[?:1.8.0_302]
> Jul 29 21:41:15 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_302]
> Jul 29 21:41:15 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_302]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-23575) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails with AsynchronousException
[
https://issues.apache.org/jira/browse/FLINK-23575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz closed FLINK-23575.
Resolution: Duplicate
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails with
> AsynchronousException
> --
>
> Key: FLINK-23575
> URL: https://issues.apache.org/jira/browse/FLINK-23575
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21246&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=4938
> {code}
> Jul 31 21:32:59 [ERROR] Tests run: 72, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 655.588 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase
> Jul 31 21:32:59 [ERROR] shouldRescaleUnalignedCheckpoint[upscale pipeline
> from 20 to 21, buffersPerChannel =
> 1](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 21.288 s <<< ERROR!
> Jul 31 21:32:59 org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> Jul 31 21:32:59 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> Jul 31 21:32:59 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:159)
> Jul 31 21:32:59 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:543)
> Jul 31 21:32:59 at
> sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
> Jul 31 21:32:59 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 31 21:32:59 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 31 21:32:59 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 31 21:32:59 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 31 21:32:59 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 31 21:32:59 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 31 21:32:59 at org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> Jul 31 21:32:59 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Jul 31 21:32:59 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jul 31 21:32:59 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 31 21:32:59 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 31 21:32:59 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 31 21:32:59 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 31 21:32:59 at org.junit.runners.Suite.runChild(Suite.java:128)
> Jul 31 21:32:59 at org.junit.runners.Suite.runChild(Suite.java:27)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 31 21:32:59 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 31 21:32:59 at
> org.apache.maven.suref
[jira] [Updated] (FLINK-23590) StreamTaskTest#testProcessWithUnAvailableInput is flaky
[
https://issues.apache.org/jira/browse/FLINK-23590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-23590:
---
Priority: Critical (was: Major)
> StreamTaskTest#testProcessWithUnAvailableInput is flaky
> ---
>
> Key: FLINK-23590
> URL: https://issues.apache.org/jira/browse/FLINK-23590
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0
>Reporter: David Morávek
>Priority: Critical
> Fix For: 1.14.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21218&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb]
>
> {code:java}
> java.lang.AssertionError:
> Expected: a value equal to or greater than <22L>
> but: <217391L> was less than <22L> at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:964)
> at org.junit.Assert.assertThat(Assert.java:930)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testProcessWithUnAvailableInput(StreamTaskTest.java:1561)
> at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.lang.Thread.run(Thread.java:829){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-23590) StreamTaskTest#testProcessWithUnAvailableInput is flaky
[
https://issues.apache.org/jira/browse/FLINK-23590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-23590:
--
Assignee: Anton Kalashnikov
> StreamTaskTest#testProcessWithUnAvailableInput is flaky
> ---
>
> Key: FLINK-23590
> URL: https://issues.apache.org/jira/browse/FLINK-23590
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0
>Reporter: David Morávek
>Assignee: Anton Kalashnikov
>Priority: Critical
> Fix For: 1.14.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21218&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb]
>
> {code:java}
> java.lang.AssertionError:
> Expected: a value equal to or greater than <22L>
> but: <217391L> was less than <22L> at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:964)
> at org.junit.Assert.assertThat(Assert.java:930)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testProcessWithUnAvailableInput(StreamTaskTest.java:1561)
> at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.lang.Thread.run(Thread.java:829){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23598) DataOutputSerializer.writeBytes updates position twice
nihileon created FLINK-23598: Summary: DataOutputSerializer.writeBytes updates position twice Key: FLINK-23598 URL: https://issues.apache.org/jira/browse/FLINK-23598 Project: Flink Issue Type: Bug Components: API / Core Affects Versions: 1.13.0 Reporter: nihileon Attachments: image-2021-08-03-16-07-17-790.png, image-2021-08-03-16-07-40-338.png, image-2021-08-03-16-08-09-249.png DataOutputSerializer.writeBytes updates this.position twice, which only need to be update once. If the initiate position is 0 and I write a string of length 10, the position will be updated to 20. !image-2021-08-03-16-07-17-790.png|width=762,height=372!!image-2021-08-03-16-07-40-338.png|width=744,height=166! !image-2021-08-03-16-08-09-249.png|width=698,height=269! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23598) DataOutputSerializer.writeBytes updates position twice
[ https://issues.apache.org/jira/browse/FLINK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nihileon updated FLINK-23598: - Priority: Critical (was: Minor) > DataOutputSerializer.writeBytes updates position twice > -- > > Key: FLINK-23598 > URL: https://issues.apache.org/jira/browse/FLINK-23598 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.13.0 >Reporter: nihileon >Priority: Critical > Attachments: image-2021-08-03-16-07-17-790.png, > image-2021-08-03-16-07-40-338.png, image-2021-08-03-16-08-09-249.png > > Original Estimate: 10m > Remaining Estimate: 10m > > DataOutputSerializer.writeBytes updates this.position twice, which only need > to be update once. > If the initiate position is 0 and I write a string of length 10, the position > will be updated to 20. > !image-2021-08-03-16-07-17-790.png|width=762,height=372!!image-2021-08-03-16-07-40-338.png|width=744,height=166! > !image-2021-08-03-16-08-09-249.png|width=698,height=269! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23593) Performance regression on 15.07.2021
[
https://issues.apache.org/jira/browse/FLINK-23593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392091#comment-17392091
]
Zhu Zhu commented on FLINK-23593:
-
I'd like to understand why the regression happens due to FLINK-23372 before
deciding whether it is really a problem.
I do not know much about these two benchmarks. IIUC, the mentioned benchmarks
are for batch jobs only.
After FLINK-23372, batch job upstream and downstream tasks no longer share
slots. So maybe the reason is that upstream and downstream tasks are deployed
to different task managers and cannot send data via local input channels?
[~twalthr] do you think this is the cause? or any other suspicions?
> Performance regression on 15.07.2021
>
>
> Key: FLINK-23593
> URL: https://issues.apache.org/jira/browse/FLINK-23593
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Benchmarks
>Affects Versions: 1.14.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.14.0
>
>
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedMultiInput&env=2
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedTwoInput&env=2
> {noformat}
> pnowojski@piotr-mbp: [~/flink - ((no branch, bisect started on pr/16589))] $
> git ls f4afbf3e7de..eb8100f7afe
> eb8100f7afe [4 weeks ago] (pn/bad, bad, refs/bisect/bad)
> [FLINK-22017][coordination] Allow BLOCKING result partition to be
> individually consumable [Thesharing]
> d2005268b1e [4 weeks ago] (HEAD, pn/bisect-4, bisect-4)
> [FLINK-22017][coordination] Get the ConsumedPartitionGroup that
> IntermediateResultPartition and DefaultResultPartition belong to [Thesharing]
> d8b1a6fd368 [3 weeks ago] [FLINK-23372][streaming-java] Disable
> AllVerticesInSameSlotSharingGroupByDefault in batch mode [Timo Walther]
> 4a78097d038 [3 weeks ago] (pn/bisect-3, bisect-3,
> refs/bisect/good-4a78097d0385749daceafd8326930c8cc5f26f1a)
> [FLINK-21928][clients][runtime] Introduce static method constructors of
> DuplicateJobSubmissionException for better readability. [David Moravek]
> 172b9e32215 [3 weeks ago] [FLINK-21928][clients] JobManager failover should
> succeed, when trying to resubmit already terminated job in application mode.
> [David Moravek]
> f483008db86 [3 weeks ago] [FLINK-21928][core] Introduce
> org.apache.flink.util.concurrent.FutureUtils#handleException method, that
> allows future to recover from the specied exception. [David Moravek]
> d7ac08c2ac0 [3 weeks ago] (pn/bisect-2, bisect-2,
> refs/bisect/good-d7ac08c2ac06b9ff31707f3b8f43c07817814d4f)
> [FLINK-22843][docs-zh] Document and code are inconsistent [ZhiJie Yang]
> 16c3ea427df [3 weeks ago] [hotfix] Split the final checkpoint related tests
> to a separate test class. [Yun Gao]
> 31b3d37a22c [7 weeks ago] [FLINK-21089][runtime] Skip the execution of new
> sources if finished on restore [Yun Gao]
> 20fe062e1b5 [3 weeks ago] [FLINK-21089][runtime] Skip execution for the
> legacy source task if finished on restore [Yun Gao]
> 874c627114b [3 weeks ago] [FLINK-21089][runtime] Skip the lifecycle method of
> operators if finished on restore [Yun Gao]
> ceaf24b1d88 [3 weeks ago] (pn/bisect-1, bisect-1,
> refs/bisect/good-ceaf24b1d881c2345a43f305d40435519a09cec9) [hotfix] Fix
> isClosed() for operator wrapper and proxy operator close to the operator
> chain [Yun Gao]
> 41ea591a6db [3 weeks ago] [FLINK-22627][runtime] Remove unused slot request
> protocol [Yangze Guo]
> 489346b60f8 [3 months ago] [FLINK-22627][runtime] Remove PendingSlotRequest
> [Yangze Guo]
> 8ffb4d2af36 [3 months ago] [FLINK-22627][runtime] Remove TaskManagerSlot
> [Yangze Guo]
> 72073741588 [3 months ago] [FLINK-22627][runtime] Remove SlotManagerImpl and
> its related tests [Yangze Guo]
> bdb3b7541b3 [3 months ago] [hotfix][yarn] Remove unused internal options in
> YarnConfigOptionsInternal [Yangze Guo]
> a6a9b192eac [3 weeks ago] [FLINK-23201][streaming] Reset alignment only for
> the currently processed checkpoint [Anton Kalashnikov]
> b35701a35c7 [3 weeks ago] [FLINK-23201][streaming] Calculate checkpoint
> alignment time only for last started checkpoint [Anton Kalashnikov]
> 3abec22c536 [3 weeks ago] [FLINK-23107][table-runtime] Separate
> implementation of deduplicate rank from other rank functions [Shuo Cheng]
> 1a195f5cc59 [3 weeks ago] [FLINK-16093][docs-zh] Translate "System Functions"
> page of "Functions" into Chinese (#16348) [ZhiJie Yang]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23451) FLIP-183: Dynamic buffer size adjustment (Buffer debloat)
[ https://issues.apache.org/jira/browse/FLINK-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-23451: --- Summary: FLIP-183: Dynamic buffer size adjustment (Buffer debloat) (was: FLIP-183: Dynamic buffer size adjustment) > FLIP-183: Dynamic buffer size adjustment (Buffer debloat) > - > > Key: FLINK-23451 > URL: https://issues.apache.org/jira/browse/FLINK-23451 > Project: Flink > Issue Type: New Feature >Reporter: Anton Kalashnikov >Priority: Major > > Umbrella ticket for > https://cwiki.apache.org/confluence/display/FLINK/FLIP-183%3A+Dynamic+buffer+size+adjustment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23453) Dynamic calculation of the buffer size
[ https://issues.apache.org/jira/browse/FLINK-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reassigned FLINK-23453: -- Assignee: Anton Kalashnikov > Dynamic calculation of the buffer size > -- > > Key: FLINK-23453 > URL: https://issues.apache.org/jira/browse/FLINK-23453 > Project: Flink > Issue Type: Sub-task >Reporter: Anton Kalashnikov >Assignee: Anton Kalashnikov >Priority: Major > Labels: pull-request-available > > To calculate the desired buffer size we need to take into account the > throughput, configuration(timeInInputBuffer), and the actual number of > buffers in use. It makes sense to use EMA for this calculation to smoothen > out intermittent spikes. > The calculation based on the actual number of buffers in use helps to avoid > problems with the data skew (when only a couple of channels out of thousands > have any data). So the solution needs to reliably and efficiently calculate > either the estimated or an average number of buffers in use. > Buffer size can be erratic if it’s not trivial to make it stable in the MVP. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23454) Sending the buffer of the right size for unicast
[
https://issues.apache.org/jira/browse/FLINK-23454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-23454:
--
Assignee: Anton Kalashnikov
> Sending the buffer of the right size for unicast
>
>
> Key: FLINK-23454
> URL: https://issues.apache.org/jira/browse/FLINK-23454
> Project: Flink
> Issue Type: Sub-task
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
>
> It is not enough to know just the number of available buffers (credits) for
> the downstream because the size of these buffers can be different. So we are
> proposing to resolve this problem in the following way: If the downstream
> buffer size is changed then the upstream should send the buffer of the size
> not greater than the new one regardless of how big the current buffer on the
> upstream. (pollBuffer should receive parameters like bufferSize and return
> buffer not greater than it)
> Downstream will be able to support any buffer size < max buffer size, so it
> should be just good enough to request BufferBuilder with new size after
> getting announcement, and leaving existing BufferBuilder/BufferConsumers
> unchanged. In other words code in {{PipelinedSubpartition(View)}} doesn’t
> need to be changed (apart of forwarding new buffer size to the
> {{BufferWritingResultPartition}}). All buffer size adjustments can be
> implemented exclusively in {{BufferWritingResultPartition}}.
> If different downstream subtasks have different throughput and hence
> different desired buffer sizes, then a single upstream subtask has to support
> having two different subpartitions with different buffer sizes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #16679: [FLINK-23402][streaming-java] Simplify shuffle mode for batch execution
zhuzhurk commented on a change in pull request #16679:
URL: https://github.com/apache/flink/pull/16679#discussion_r681547939
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/BatchShuffleMode.java
##
@@ -41,7 +41,7 @@
* to run upstream and downstream tasks simultaneously.
*/
@PublicEvolving
-public enum ShuffleMode implements DescribedEnum {
+public enum BatchShuffleMode implements DescribedEnum {
Review comment:
I agree the information can be useful for understanding.
I was a bit confused because `streaming`/`batch` modes stands for
`streaming`/`batch` jobs. But this enum only works for batch jobs.
So maybe explain it against `PIPELINED`/`BLOCKING` exchanges instead of
`streaming`/`batch` modes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-23599) Remove JobVertex#connectIdInput
Zhilong Hong created FLINK-23599:
Summary: Remove JobVertex#connectIdInput
Key: FLINK-23599
URL: https://issues.apache.org/jira/browse/FLINK-23599
Project: Flink
Issue Type: Technical Debt
Components: Runtime / Coordination
Affects Versions: 1.14.0
Reporter: Zhilong Hong
Fix For: 1.14.0
{{JobVertex#connectIdInput}} is not used in production anymore. It's only used
in the unit tests {{testAttachViaIds}} and {{testCannotConnectMissingId}}
located in {{DefaultExecutionGraphConstructionTest}}. However, these two test
cases are designed to test this method. Therefore, this method and its test
cases can be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-23599) Remove JobVertex#connectIdInput
[
https://issues.apache.org/jira/browse/FLINK-23599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu reassigned FLINK-23599:
---
Assignee: Zhilong Hong
> Remove JobVertex#connectIdInput
> ---
>
> Key: FLINK-23599
> URL: https://issues.apache.org/jira/browse/FLINK-23599
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Zhilong Hong
>Assignee: Zhilong Hong
>Priority: Major
> Fix For: 1.14.0
>
>
> {{JobVertex#connectIdInput}} is not used in production anymore. It's only
> used in the unit tests {{testAttachViaIds}} and
> {{testCannotConnectMissingId}} located in
> {{DefaultExecutionGraphConstructionTest}}. However, these two test cases are
> designed to test this method. Therefore, this method and its test cases can
> be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan opened a new pull request #16685: [FLINK-23279][tests] Randomly use Changelog Backend in tests
rkhachatryan opened a new pull request #16685: URL: https://github.com/apache/flink/pull/16685 ## What is the purpose of the change Randomize whether ChangelogStateBackend is used in tests. The option was added previously (#16290) but it wasn't possible to turn it on before merging DFS changelog writer. The adjustments are: - use FS DSTL in some tests with heavy load - don't discard changelog private state to avoid snapshot invalidation on abortion - will be addressed in FLINK-23139 - don't use changelog if local recovery is enabled (which is not supported ATM) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liyubin117 commented on pull request #16604: [FLINK-23499][Runtime][Configuration]Eliminate the influence of scheme on splitting paths
liyubin117 commented on pull request #16604: URL: https://github.com/apache/flink/pull/16604#issuecomment-891648262 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liyubin117 removed a comment on pull request #16604: [FLINK-23499][Runtime][Configuration]Eliminate the influence of scheme on splitting paths
liyubin117 removed a comment on pull request #16604: URL: https://github.com/apache/flink/pull/16604#issuecomment-891648262 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23279) Enable changelog backend in tests
[ https://issues.apache.org/jira/browse/FLINK-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-23279: --- Labels: pull-request-available (was: ) > Enable changelog backend in tests > - > > Key: FLINK-23279 > URL: https://issues.apache.org/jira/browse/FLINK-23279 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > FLINK-21448 adds the capability (test randomization), but it can't be turned > on as there are some test failures: FLINK-23276, FLINK-23277, FLINK-23278 > (should be enabled after those bugs fixed).. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] yulei0824 commented on pull request #16645: [FLINK-23539][metrics-influxdb] InfluxDBReporter should filter charac…
yulei0824 commented on pull request #16645: URL: https://github.com/apache/flink/pull/16645#issuecomment-891649410 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15304: [FLINK-20731][connector] Introduce Pulsar Source
flinkbot edited a comment on pull request #15304: URL: https://github.com/apache/flink/pull/15304#issuecomment-803430086 ## CI report: * 04da1862ec1f107c74017283c916229b560d9731 UNKNOWN * 4dcee9f07135401160aae3f0d01bd480630f808f UNKNOWN * d326abc93378bb3c4b53616e7717c409d9876ade UNKNOWN * 2af90ed68fc1cf845251e606ced17c9017b8f165 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21364) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16685: [FLINK-23279][tests] Randomly use Changelog Backend in tests
flinkbot commented on pull request #16685: URL: https://github.com/apache/flink/pull/16685#issuecomment-891651007 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 32807f9693e4b834d8f89b0e109d72679de33add (Tue Aug 03 08:34:50 UTC 2021) **Warnings:** * **2 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16368: ignore
flinkbot edited a comment on pull request #16368: URL: https://github.com/apache/flink/pull/16368#issuecomment-873854539 ## CI report: * c8f4d8038e345ae3083ce5c32d5f54ecd812792b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21350) * 33ad09863a81711ff79b040e12eabb051470ba40 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23598) DataOutputSerializer.writeBytes updates position twice
[
https://issues.apache.org/jira/browse/FLINK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392169#comment-17392169
]
Stephan Ewen commented on FLINK-23598:
--
You are right, this looks wrong. Do you want to create a fix for this (with
test)?
The {{writeBytes(String)}} method is fortunately never used in Flink, because
it is strange (does not encode length) and also does not have a corresponding
read method.
> DataOutputSerializer.writeBytes updates position twice
> --
>
> Key: FLINK-23598
> URL: https://issues.apache.org/jira/browse/FLINK-23598
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.13.0
>Reporter: nihileon
>Priority: Critical
> Attachments: image-2021-08-03-16-07-17-790.png,
> image-2021-08-03-16-07-40-338.png, image-2021-08-03-16-08-09-249.png
>
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> DataOutputSerializer.writeBytes updates this.position twice, which only need
> to be update once.
> If the initiate position is 0 and I write a string of length 10, the position
> will be updated to 20.
> !image-2021-08-03-16-07-17-790.png|width=762,height=372!!image-2021-08-03-16-07-40-338.png|width=744,height=166!
> !image-2021-08-03-16-08-09-249.png|width=698,height=269!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16576: [FLINK-23463][doc] Replace the tag with ShortCodes in document
flinkbot edited a comment on pull request #16576: URL: https://github.com/apache/flink/pull/16576#issuecomment-885383955 ## CI report: * 3ec79c99b0f88f08f877037e72e7575db41080dc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21351) * e953be8b85c3dfc68e3965cff8a819189cd20eee Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21369) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16630: [FLINK-23531][table]Allow skip all change log for row-time deduplicate mini-batch
flinkbot edited a comment on pull request #16630: URL: https://github.com/apache/flink/pull/16630#issuecomment-17597 ## CI report: * dd3f5c216993d99c6742add9f4e878a54b52343a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21366) * c588afeb471bcd8c82ffa9149663b1f4b1084b6b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16640: [FLINK-22891][runtime] Using CompletableFuture to sync the scheduling…
flinkbot edited a comment on pull request #16640: URL: https://github.com/apache/flink/pull/16640#issuecomment-889598196 ## CI report: * ed73065a1c804251ca5731fa860dadd4fcf7d16d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21348) * 81126114e90f8947dba92e4bc1bf785f6fc86ac5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21370) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16645: [FLINK-23539][metrics-influxdb] InfluxDBReporter should filter charac…
flinkbot edited a comment on pull request #16645: URL: https://github.com/apache/flink/pull/16645#issuecomment-889643808 ## CI report: * 438b91bf2081a34d4a928cabb45454441386972e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21232) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21220) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21375) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16667: [FLINK-22912][python] Support state ttl in Python DataStream API
flinkbot edited a comment on pull request #16667: URL: https://github.com/apache/flink/pull/16667#issuecomment-890801858 ## CI report: * 207ff3a175da76f1e0878aa0a9788dbe478c83f0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21343) * eca746e23bdf0ebfaeba1c5a14ca50262241dec5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21371) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16684: FIX FLINK-23580
flinkbot commented on pull request #16684: URL: https://github.com/apache/flink/pull/16684#issuecomment-891654033 ## CI report: * 5493c026a97540ffc65abe158545e0236b4d9e39 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #16679: [FLINK-23402][streaming-java] Simplify shuffle mode for batch execution
twalthr commented on a change in pull request #16679:
URL: https://github.com/apache/flink/pull/16679#discussion_r681556886
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/BatchShuffleMode.java
##
@@ -41,7 +41,7 @@
* to run upstream and downstream tasks simultaneously.
*/
@PublicEvolving
-public enum ShuffleMode implements DescribedEnum {
+public enum BatchShuffleMode implements DescribedEnum {
Review comment:
I removed the mentioning of streaming/batch mode in the enum values. But
I would leave the more extensive class JavaDoc to understand the meaning of
`BatchShuffleMode` in general and why it doesn't apply to streaming mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Thesharing closed pull request #15382: [FLINK-21915] Optimize Execution#finishPartitionsAndUpdateConsumers
Thesharing closed pull request #15382: URL: https://github.com/apache/flink/pull/15382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Thesharing commented on pull request #15382: [FLINK-21915] Optimize Execution#finishPartitionsAndUpdateConsumers
Thesharing commented on pull request #15382: URL: https://github.com/apache/flink/pull/15382#issuecomment-891657709 Due to FLINK-22017, now blocking partitions will be individually consumable once it's finished. finishPartitionAndUpdateConsumer will be called every partition is finished. The proposed optimization is no longer valid. Furthermore, this function is called only when there is intra-region edges in the graph. In this case the downstream vertices are DEPLOYING/RUNNING when the upstream vertices are FINISHED. The scenario is rare. Thus for now we just close it. If there's a new idea about it, we'd like to reopen it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23593) Performance regression on 15.07.2021
[
https://issues.apache.org/jira/browse/FLINK-23593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392173#comment-17392173
]
Stephan Ewen commented on FLINK-23593:
--
I would guess that with tasks no longer sharing the slot, some stages that were
previously parallel are not serial?
> Performance regression on 15.07.2021
>
>
> Key: FLINK-23593
> URL: https://issues.apache.org/jira/browse/FLINK-23593
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Benchmarks
>Affects Versions: 1.14.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.14.0
>
>
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedMultiInput&env=2
> http://codespeed.dak8s.net:8000/timeline/?ben=sortedTwoInput&env=2
> {noformat}
> pnowojski@piotr-mbp: [~/flink - ((no branch, bisect started on pr/16589))] $
> git ls f4afbf3e7de..eb8100f7afe
> eb8100f7afe [4 weeks ago] (pn/bad, bad, refs/bisect/bad)
> [FLINK-22017][coordination] Allow BLOCKING result partition to be
> individually consumable [Thesharing]
> d2005268b1e [4 weeks ago] (HEAD, pn/bisect-4, bisect-4)
> [FLINK-22017][coordination] Get the ConsumedPartitionGroup that
> IntermediateResultPartition and DefaultResultPartition belong to [Thesharing]
> d8b1a6fd368 [3 weeks ago] [FLINK-23372][streaming-java] Disable
> AllVerticesInSameSlotSharingGroupByDefault in batch mode [Timo Walther]
> 4a78097d038 [3 weeks ago] (pn/bisect-3, bisect-3,
> refs/bisect/good-4a78097d0385749daceafd8326930c8cc5f26f1a)
> [FLINK-21928][clients][runtime] Introduce static method constructors of
> DuplicateJobSubmissionException for better readability. [David Moravek]
> 172b9e32215 [3 weeks ago] [FLINK-21928][clients] JobManager failover should
> succeed, when trying to resubmit already terminated job in application mode.
> [David Moravek]
> f483008db86 [3 weeks ago] [FLINK-21928][core] Introduce
> org.apache.flink.util.concurrent.FutureUtils#handleException method, that
> allows future to recover from the specied exception. [David Moravek]
> d7ac08c2ac0 [3 weeks ago] (pn/bisect-2, bisect-2,
> refs/bisect/good-d7ac08c2ac06b9ff31707f3b8f43c07817814d4f)
> [FLINK-22843][docs-zh] Document and code are inconsistent [ZhiJie Yang]
> 16c3ea427df [3 weeks ago] [hotfix] Split the final checkpoint related tests
> to a separate test class. [Yun Gao]
> 31b3d37a22c [7 weeks ago] [FLINK-21089][runtime] Skip the execution of new
> sources if finished on restore [Yun Gao]
> 20fe062e1b5 [3 weeks ago] [FLINK-21089][runtime] Skip execution for the
> legacy source task if finished on restore [Yun Gao]
> 874c627114b [3 weeks ago] [FLINK-21089][runtime] Skip the lifecycle method of
> operators if finished on restore [Yun Gao]
> ceaf24b1d88 [3 weeks ago] (pn/bisect-1, bisect-1,
> refs/bisect/good-ceaf24b1d881c2345a43f305d40435519a09cec9) [hotfix] Fix
> isClosed() for operator wrapper and proxy operator close to the operator
> chain [Yun Gao]
> 41ea591a6db [3 weeks ago] [FLINK-22627][runtime] Remove unused slot request
> protocol [Yangze Guo]
> 489346b60f8 [3 months ago] [FLINK-22627][runtime] Remove PendingSlotRequest
> [Yangze Guo]
> 8ffb4d2af36 [3 months ago] [FLINK-22627][runtime] Remove TaskManagerSlot
> [Yangze Guo]
> 72073741588 [3 months ago] [FLINK-22627][runtime] Remove SlotManagerImpl and
> its related tests [Yangze Guo]
> bdb3b7541b3 [3 months ago] [hotfix][yarn] Remove unused internal options in
> YarnConfigOptionsInternal [Yangze Guo]
> a6a9b192eac [3 weeks ago] [FLINK-23201][streaming] Reset alignment only for
> the currently processed checkpoint [Anton Kalashnikov]
> b35701a35c7 [3 weeks ago] [FLINK-23201][streaming] Calculate checkpoint
> alignment time only for last started checkpoint [Anton Kalashnikov]
> 3abec22c536 [3 weeks ago] [FLINK-23107][table-runtime] Separate
> implementation of deduplicate rank from other rank functions [Shuo Cheng]
> 1a195f5cc59 [3 weeks ago] [FLINK-16093][docs-zh] Translate "System Functions"
> page of "Functions" into Chinese (#16348) [ZhiJie Yang]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23372) Disable AllVerticesInSameSlotSharingGroupByDefault in DataStream batch mode
[
https://issues.apache.org/jira/browse/FLINK-23372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392174#comment-17392174
]
Stephan Ewen commented on FLINK-23372:
--
This is a pretty foundational change and decision here. It may make sense, but
can we outline the pros/cons here to make that decision transparent?
> Disable AllVerticesInSameSlotSharingGroupByDefault in DataStream batch mode
> ---
>
> Key: FLINK-23372
> URL: https://issues.apache.org/jira/browse/FLINK-23372
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> In order to unify the behavior of DataStream API and Table API batch mode, we
> should disable AllVerticesInSameSlotSharingGroupByDefault also in DataStream
> API.
> FLINK-20001 reverted setting this flag but without concrete arguments and the
> following comment: {{reconsider actually setting this flag in the future}}
> After a offline chat with [~zhuzh], we should introduce this again for
> consistency:
> {code}
> The goal to assign different regions to different slot sharing groups by
> default is to reduce waste of resources. In batch jobs, there can be one
> region which has data dependency on another region. And the resource
> computation for slots and managed memory will be affected:
> 1. If these regions are in the same slot sharing group, the group will
> require a large slot which can host tasks from both the regions.
> 2. In managed memory fraction computing, tasks from both regions will be
> considered to compete for managed memory, so each task will be assigned with
> a smaller managed memory fraction (FLIP-53).
> However, those regions will not run at the same time and results in a waste
> of resources.
> For streaming jobs, all tasks will run at the same time. So assigning them to
> the same slot sharing group will not result resource waste.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23600) Rework StateFun's remote module parsing and binding
Tzu-Li (Gordon) Tai created FLINK-23600:
---
Summary: Rework StateFun's remote module parsing and binding
Key: FLINK-23600
URL: https://issues.apache.org/jira/browse/FLINK-23600
Project: Flink
Issue Type: New Feature
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Currently, we have a {{JsonModule}} class that is responsible for parsing
user's module YAML specifications, resolving the specification into application
components (i.e. function providers, ingresses, routers, and egresses) that is
then bound to the application universe.
Over time, the {{JsonModule}} class has overgrown with several changes as we
progressively adapted the YAML format.
* The class handles ALL kinds of components, including ingresses / functions /
egresses etc. The code is extremely fragile and becoming hard to extend.
* Users have no access to extend this class, if they somehow need to plugin
custom components (e.g. adding an unsupported ingress / egress, custom protocol
implementations etc).
We aim to rework this with the following goals in mind:
# The system should only handle {{module.yaml}} parsing up to the point where
it extracts a list of JSON objects that individually represent an application
component.
# The system has no knowledge of what each JSON objects contains, other than
its {{TypeName}} which would map to a corresponding {{ComponentBinder}}.
# A {{ComponentBinder}} is essentially an extension bound to the system that
knows how to parse a specific JSON object, and bind components to the
application universe.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] twalthr commented on pull request #16678: [FLINK-15804][table-api-scala] Use new type inference for implicit ScalarFunction calls in Scala
twalthr commented on pull request #16678: URL: https://github.com/apache/flink/pull/16678#issuecomment-891662074 Thanks @Airblader. I agree that the default function identifier is hard to read. But this is a followup issue see FLINK-15585. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Thesharing opened a new pull request #16686: [FLINK-23599] Remove JobVertex#connectIdInput
Thesharing opened a new pull request #16686: URL: https://github.com/apache/flink/pull/16686 ## What is the purpose of the change *JobVertex#connectIdInput is not used in production anymore. It's only used in the unit tests testAttachViaIds and testCannotConnectMissingId located in DefaultExecutionGraphConstructionTest. However, these two test cases are designed to test this method. Therefore, this method and its test cases can be removed. For more details please check FLINK-23599.* ## Brief change log - *Remove JobVertex#connectIdInput* - *Remove DefaultExecutionGraphConstructionTest#testAttachViaIds and testCannotConnectMissingId* - *Fix checkstyle in JobVertex.java and DefaultExecutionGraphConstructionTest.java* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**yes** / no / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23599) Remove JobVertex#connectIdInput
[
https://issues.apache.org/jira/browse/FLINK-23599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-23599:
---
Labels: pull-request-available (was: )
> Remove JobVertex#connectIdInput
> ---
>
> Key: FLINK-23599
> URL: https://issues.apache.org/jira/browse/FLINK-23599
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Zhilong Hong
>Assignee: Zhilong Hong
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> {{JobVertex#connectIdInput}} is not used in production anymore. It's only
> used in the unit tests {{testAttachViaIds}} and
> {{testCannotConnectMissingId}} located in
> {{DefaultExecutionGraphConstructionTest}}. However, these two test cases are
> designed to test this method. Therefore, this method and its test cases can
> be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] Thesharing closed pull request #15387: [FLINK-21920] Optimize DefaultScheduler#allocateSlots
Thesharing closed pull request #15387: URL: https://github.com/apache/flink/pull/15387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Thesharing commented on pull request #15387: [FLINK-21920] Optimize DefaultScheduler#allocateSlots
Thesharing commented on pull request #15387: URL: https://github.com/apache/flink/pull/15387#issuecomment-891664350 The proposed solution will make ExecutionGraphToInputsLocationsRetrieverAdapter stateful, which is hard to maintain. However, the improvement of this optimization seems not obvious. For now we just close it. If later we got a better idea, we'd like to reopen it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
