[
https://issues.apache.org/jira/browse/FLINK-39968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yyka updated FLINK-39968:
-------------------------
Description:
h2. Background & Problem Analysis
* *{{MemorySegment}} core usage scenarios*
{{MemorySegment}} is the core memory abstraction of Flink, which has two key
usage scenarios:
*
** {*}Memory buffer pool{*}: Manages reusable memory blocks (network buffers,
managed memory), requiring explicit {{free()}} to reclaim memory;
** {*}In-operator-chain data copy transmission{*}: When the pipeline
configuration {{pipeline.object-reuse = false}} (default off), a large number
of temporary {{MemorySegment}} instances are created for data copy transmission
between operators, and these instances *never call the {{free()}} method*
during their lifecycle.
* *Redundant object allocation overhead*
The current {{MemorySegment}} unconditionally initializes a {{final
AtomicBoolean isFreedAtomic}} in the constructor:
*this.isFreedAtomic = new AtomicBoolean(false);*
In the in-operator-chain data copy transmission (the default scenario
with massive temporary segment creation), {{isFreedAtomic}} is never used,
resulting in meaningless object allocation;
Massive creation of temporary {{AtomicBoolean}} objects increases GC
pressure and constructor execution time, which directly affects the throughput
of high-concurrent streaming jobs.
h2. Optimization Plan
# Replace the member variable {{final AtomicBoolean isFreedAtomic}} with a
primitive {{{}boolean isFreed{}}};
# Add the {{synchronized}} modifier to the {{free()}} method to ensure the
atomicity and thread safety of the free state judgment and modification;
# Retain all original functional logic of the {{free()}} method, only optimize
the underlying state storage and thread safety implementation.
h2. Benchmark Test Results
# {*}Micro-benchmark (new MemorySegment){*}: Using primitive boolean instead
of {{new AtomicBoolean}} reduces constructor overhead, with *3% performance
improvement* for single instance creation;
# {*}End-to-end benchmark (Nexmark){*}: The overall job throughput is
improved, and GC frequency/pause time is reduced, verifying the effectiveness
of the optimization in real streaming scenarios.
was:
h2. Background & Problem Analysis
# *{{MemorySegment}} core usage scenarios*
{{MemorySegment}} is the core memory abstraction of Flink, which has two key
usage scenarios:
** {*}Memory buffer pool{*}: Manages reusable memory blocks (network buffers,
managed memory), requiring explicit {{free()}} to reclaim memory;
** {*}In-operator-chain data copy transmission{*}: When the pipeline
configuration {{pipeline.object-reuse = false}} (default off), a large number
of temporary {{MemorySegment}} instances are created for data copy transmission
between operators, and these instances *never call the {{free()}} method*
during their lifecycle.
# *Redundant object allocation overhead*
The current {{MemorySegment}} unconditionally initializes a {{final
AtomicBoolean isFreedAtomic}} in the constructor:
*this.isFreedAtomic = new AtomicBoolean(false);*
In the in-operator-chain data copy transmission (the default scenario with
massive temporary segment creation), {{isFreedAtomic}} is never used, resulting
in meaningless object allocation;
Massive creation of temporary {{AtomicBoolean}} objects increases GC pressure
and constructor execution time, which directly affects the throughput of
high-concurrent streaming jobs.
h2. Optimization Plan
# Replace the member variable {{final AtomicBoolean isFreedAtomic}} with a
primitive {{{}boolean isFreed{}}};
# Add the {{synchronized}} modifier to the {{free()}} method to ensure the
atomicity and thread safety of the free state judgment and modification;
# Retain all original functional logic of the {{free()}} method, only optimize
the underlying state storage and thread safety implementation.
h2. Benchmark Test Results
# {*}Micro-benchmark (new MemorySegment){*}: Using primitive boolean instead
of {{new AtomicBoolean}} reduces constructor overhead, with *3% performance
improvement* for single instance creation;
# {*}End-to-end benchmark (Nexmark){*}: The overall job throughput is
improved, and GC frequency/pause time is reduced, verifying the effectiveness
of the optimization in real streaming scenarios.
> [Optimization] Reduce redundant AtomicBoolean allocation in MemorySegment to
> improve construction performance
> -------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-39968
> URL: https://issues.apache.org/jira/browse/FLINK-39968
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
> Reporter: yyka
> Priority: Major
> Labels: performance
>
> h2. Background & Problem Analysis
> * *{{MemorySegment}} core usage scenarios*
> {{MemorySegment}} is the core memory abstraction of Flink, which has two key
> usage scenarios:
> *
> ** {*}Memory buffer pool{*}: Manages reusable memory blocks (network
> buffers, managed memory), requiring explicit {{free()}} to reclaim memory;
> ** {*}In-operator-chain data copy transmission{*}: When the pipeline
> configuration {{pipeline.object-reuse = false}} (default off), a large number
> of temporary {{MemorySegment}} instances are created for data copy
> transmission between operators, and these instances *never call the
> {{free()}} method* during their lifecycle.
> * *Redundant object allocation overhead*
> The current {{MemorySegment}} unconditionally initializes a {{final
> AtomicBoolean isFreedAtomic}} in the constructor:
> *this.isFreedAtomic = new AtomicBoolean(false);*
> In the in-operator-chain data copy transmission (the default
> scenario with massive temporary segment creation), {{isFreedAtomic}} is never
> used, resulting in meaningless object allocation;
> Massive creation of temporary {{AtomicBoolean}} objects increases
> GC pressure and constructor execution time, which directly affects the
> throughput of high-concurrent streaming jobs.
> h2. Optimization Plan
> # Replace the member variable {{final AtomicBoolean isFreedAtomic}} with a
> primitive {{{}boolean isFreed{}}};
> # Add the {{synchronized}} modifier to the {{free()}} method to ensure the
> atomicity and thread safety of the free state judgment and modification;
> # Retain all original functional logic of the {{free()}} method, only
> optimize the underlying state storage and thread safety implementation.
> h2. Benchmark Test Results
> # {*}Micro-benchmark (new MemorySegment){*}: Using primitive boolean instead
> of {{new AtomicBoolean}} reduces constructor overhead, with *3% performance
> improvement* for single instance creation;
> # {*}End-to-end benchmark (Nexmark){*}: The overall job throughput is
> improved, and GC frequency/pause time is reduced, verifying the effectiveness
> of the optimization in real streaming scenarios.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
