[
https://issues.apache.org/jira/browse/KAFKA-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744760#comment-16744760
]
ASF GitHub Bot commented on KAFKA-7652:
---------------------------------------
guozhangwang commented on pull request #6161: KAFKA-7652: Part II; Add
single-point query for SessionStore and use for flushing / getter
URL: https://github.com/apache/kafka/pull/6161
https://github.com/apache/kafka/pull/2972 tried to fix a bug about flushing
operation, but it was not complete, since `findSessions(key, earliestEnd,
latestStart)` does not guarantee to only return a single entry since its
semantics are to return any sessions whose end > earliestEnd and whose start <
latestStart.
I've tried various ways to fix it completely and I ended up having to add a
single-point query to the public ReadOnlySessionStore API for the exact needed
semantics. It is used for flushing to read the old values (otherwise the wrong
old values will be sent downstreams, hence it is a correctness issue) and also
for getting the value for value-getters (it is for perf only).
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kafka Streams Session store performance degradation from 0.10.2.2 to 0.11.0.0
> -----------------------------------------------------------------------------
>
> Key: KAFKA-7652
> URL: https://issues.apache.org/jira/browse/KAFKA-7652
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2, 0.11.0.3, 1.1.1, 2.0.0,
> 2.0.1
> Reporter: Jonathan Gordon
> Priority: Major
> Attachments: kafka_10_2_1_flushes.txt, kafka_11_0_3_flushes.txt
>
>
> I'm creating this issue in response to [~guozhang]'s request on the mailing
> list:
> [https://lists.apache.org/thread.html/97d620f4fd76be070ca4e2c70e2fda53cafe051e8fc4505dbcca0321@%3Cusers.kafka.apache.org%3E]
> We are attempting to upgrade our Kafka Streams application from 0.10.2.1 but
> experience a severe performance degradation. The highest amount of CPU time
> seems spent in retrieving from the local cache. Here's an example thread
> profile with 0.11.0.0:
> [https://i.imgur.com/l5VEsC2.png]
> When things are running smoothly we're gated by retrieving from the state
> store with acceptable performance. Here's an example thread profile with
> 0.10.2.1:
> [https://i.imgur.com/IHxC2cZ.png]
> Some investigation reveals that it appears we're performing about 3 orders
> magnitude more lookups on the NamedCache over a comparable time period. I've
> attached logs of the NamedCache flush logs for 0.10.2.1 and 0.11.0.3.
> We're using session windows and have the app configured for
> commit.interval.ms = 30 * 1000 and cache.max.bytes.buffering = 10485760
> I'm happy to share more details if they would be helpful. Also happy to run
> tests on our data.
> I also found this issue, which seems like it may be related:
> https://issues.apache.org/jira/browse/KAFKA-4904
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
