[
https://issues.apache.org/jira/browse/FLINK-40789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141899#comment-18141899
]
Naveen Joseph commented on FLINK-40789:
---------------------------------------
[~martijnvisser] I created a separate Apache Flink fork and configured an Azure
self-hosted agent to investigate and reproduce the issue independently.
As part of the investigation, I executed the same test groups in two additional
environments:
# *Azure self-hosted agent:* The jobs completed faster compared to the
original Apache Flink Azure Pipeline.
|Job group |Flink Azure agent |Fork run |
|core |1h 03m 23s |36m 56s |
|python |1h 24m 55s |33m 21s |
|table |54m 35s |40m 32s |
|connect |35m 11s |24m 09s |
|tests |56m 30s |49m 00s |
|misc |41m 13s |29m 37s |
# *KVM environment:* Executed the same test groups on a virtual machine with 4
CPU cores and 16 GB of RAM.
All test runs passed successfully in both environments. The Azure self-hosted
agent completed the jobs faster than the original Flink Azure Pipeline, while
the KVM machine took comparatively longer despite having 4 CPU cores and 16 GB
of RAM. I also observed that these Flink test groups consume considerable CPU
and memory resources during execution.
!image-2026-11-10-15-51-43-575.png!
Based on these observations, we confirmed that a minimum of 4 CPU cores and
approximately 14 GB of memory is sufficient for running these test groups in my
KVM environment. I believe that the five self-hosted agents running on the same
machine may be competing for CPU and memory resources, potentially causing
resource contention and contributing to the intermittent failures in the
original Apache Flink Azure Pipeline.
> RescaleTimelineITCase.testRecordRescaleForNewAvailableResource records three
> rescales instead of two
> ----------------------------------------------------------------------------------------------------
>
> Key: FLINK-40789
> URL: https://issues.apache.org/jira/browse/FLINK-40789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 2.3.1, 2.4.0, 2.5.0
> Reporter: Martijn Visser
> Assignee: Naveen Joseph
> Priority: Critical
> Labels: test-stability
> Attachments: image-2026-11-10-15-51-43-575.png
>
>
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=79368&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1
> (master push 2026-09-23, test_ci core, JDK 17)
> {code}
> [ERROR]
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.testRecordRescaleForNewAvailableResource
> -- Time elapsed: 0.722 s <<< FAILURE!
> java.lang.AssertionError:
> Expected size: 2 but was: 3 in:
> [Rescale{... rescaleAttemptId=3, ... preRescaleParallelism=4,
> postRescaleParallelism=6 ... triggerCause=NEW_RESOURCE_AVAILABLE,
> terminalState=COMPLETED ...},
> Rescale{... rescaleAttemptId=2, ... preRescaleParallelism=2,
> postRescaleParallelism=4 ... triggerCause=NEW_RESOURCE_AVAILABLE,
> terminalState=COMPLETED ...},
> Rescale{... rescaleAttemptId=1, ... preRescaleParallelism=null,
> postRescaleParallelism=2 ... triggerCause=INITIAL_SCHEDULE,
> terminalState=COMPLETED ...}]
> at
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.lambda$testRecordRescaleForNewAvailableResource$2(RescaleTimelineITCase.java:170)
> at
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.runAdaptedParameterizedAssertion(RescaleTimelineITCase.java:397)
> at
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.testRecordRescaleForNewAvailableResource(RescaleTimelineITCase.java:164)
> {code}
>
>
> The test expects the initial schedule at parallelism 4 and
> one upscale to 6 after the third TaskManager starts. Here the job was first
> scheduled at 2 and went 2 -> 4 -> 6. The other RescaleTimelineITCase methods
> had similar races fixed in FLINK-40010, FLINK-40067 and FLINK-40076; this
> method has no ticket.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
