[
https://issues.apache.org/jira/browse/FLINK-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559482#comment-16559482
]
ASF GitHub Bot commented on FLINK-9981:
---------------------------------------
StefanRRichter opened a new pull request #6438: [FLINK-9981] Tune performance
of RocksDB implementation
URL: https://github.com/apache/flink/pull/6438
## What is the purpose of the change
This PR does some general performance tuning for the RocksDB based
implementation of timer state, as well as some polishing and additional tests.
Core performance improvements come from having a cache that is based on a
min-max-heap-set instead of an avl-tree. This enables us to drop the fastutil
dependency. Furthermore, we track a lower bound of the first element in the
RocksDB based store to improve iterator seek performance dramatically for cache
refills. This improvement let's us tune down the cache size by an order of
magnitude and to drop the need for the potentially dangerous and complex
bulk-pull optimization altogether.
Adds an additional test that the configuration of the state implementation
is respected by the RocksDB backend and uses RocksDB based timers for one case
in the `EventTimeWindowCheckpointingITCase`.
## Verifying this change
Via the new tests and existing coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes)
- The serializers: (yes)
- The runtime per-record code paths (performance sensitive): (yes)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Tune performance of RocksDB implementation
> ------------------------------------------
>
> Key: FLINK-9981
> URL: https://issues.apache.org/jira/browse/FLINK-9981
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
> Affects Versions: 1.6.0
> Reporter: Stefan Richter
> Assignee: Stefan Richter
> Priority: Major
> Labels: pull-request-available
>
> General performance tuning/polishing for the RocksDB implementation. We can
> figure out how caching/seeking can be improved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
