Jinzhong Li created FLINK-33881:
-----------------------------------
Summary: [TtlListState]Avoid copy and update value when
TtlListState don‘t have expired element
Key: FLINK-33881
URL: https://issues.apache.org/jira/browse/FLINK-33881
Project: Flink
Issue Type: Improvement
Components: Runtime / State Backends
Reporter: Jinzhong Li
Attachments: image-2023-12-19-21-25-21-446.png,
image-2023-12-19-21-26-43-518.png
In some scenarios, 'TtlListState#getUnexpiredOrNull ->
elementSerializer.copy(ttlValue)' consumes a lot of cpu resources.
!image-2023-12-19-21-25-21-446.png|width=444,height=100!
I found that for TtlListState#getUnexpiredOrNull， if none of the elements have
expired, it still needs to copy all the elements and update the whole list/map
in TtlIncrementalCleanup#runCleanup();
!image-2023-12-19-21-26-43-518.png|width=317,height=167!
I think we could optimize TtlListState#getUnexpiredOrNull by:
1）find the first expired element index in the list;
2）If not found, return to the original list;
3）If found, then constrct the unexpire list (puts the previous elements into
the list), and go through the subsequent elements, adding expired elements into
the list.
{code:java}
public List<TtlValue<T>> getUnexpiredOrNull(@Nonnull List<TtlValue<T>>
ttlValues) {
//.......
int firstExpireIndex = -1;
for (int i = 0; i < ttlValues.size(); i++) {
if (TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
firstExpireIndex = i;
break;
}
}
if (firstExpireIndex == -1) {
return ttlValues; //return the original ttlValues
}
List<TtlValue<T>> unexpired = new ArrayList<>(ttlValues.size());
for (int i = 0; i < ttlValues.size(); i++) {
if (i < firstExpireIndex) {
unexpired.add(ttlValues.get(i));
}
if (i > firstExpireIndex) {
if (!TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
unexpired.add(ttlValues.get(i));
}
}
}
// .....
} {code}
In this way, the extra iteration overhead is actually very very small, but the
benefit when there are no expired elements is significant.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
