gaurav-narula closed pull request #15925: KAFKA-9401 Reduce contention for
Fetch requests
URL: https://github.com/apache/kafka/pull/15925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
gaurav-narula commented on PR #15925:
URL: https://github.com/apache/kafka/pull/15925#issuecomment-2105946195
Thanks for the feedback! My bad, I wasn't aware about the specifics of the
backporting policy. Closing this and updating the JIRA.
--
This is an automated message from the Apache
ijuma commented on PR #15925:
URL: https://github.com/apache/kafka/pull/15925#issuecomment-2105944275
We don't usually cherry pick changes like this to older branches. It's not a
regression and it hasn't been released yet (hence the risk is higher).
--
This is an automated message from
gaurav-narula commented on PR #15925:
URL: https://github.com/apache/kafka/pull/15925#issuecomment-2105943872
CC: @chia7712 @soarez
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
gaurav-narula opened a new pull request, #15925:
URL: https://github.com/apache/kafka/pull/15925
Cherry picks #15836 to 3.7.
Note:
There were merge conflicts in the following files which were resolved
```
both modified:
chia7712 commented on PR #15836:
URL: https://github.com/apache/kafka/pull/15836#issuecomment-2105931546
@gaurav-narula Please file PR for branch 3.7 if you feel this one needs to
be backport :)
--
This is an automated message from the Apache Git Service.
To respond to the message,
chia7712 merged PR #15836:
URL: https://github.com/apache/kafka/pull/15836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
gaurav-narula commented on PR #15836:
URL: https://github.com/apache/kafka/pull/15836#issuecomment-2105736540
Thanks for the review! I'm fairly convinced these failures are unrelated.
The report on Github enterprise suggests the failed tests are flakey. Please
refer the following links for
chia7712 commented on PR #15836:
URL: https://github.com/apache/kafka/pull/15836#issuecomment-2101862846
@gaurav-narula Could you take a look at those failed tests? I feel they are
unrelated to this PR, so +1
--
This is an automated message from the Apache Git Service.
To respond to the
soarez commented on PR #15836:
URL: https://github.com/apache/kafka/pull/15836#issuecomment-2098918354
`:core:test` timed out only on JDK 8 and Scala 2.12. Restarted the build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
chia7712 commented on PR #15836:
URL: https://github.com/apache/kafka/pull/15836#issuecomment-2097148532
@gaurav-narula Could you please rebase code to trigger QA again? It seems we
have thread leaks in some tests :(
--
This is an automated message from the Apache Git Service.
To respond
gaurav-narula commented on PR #15836:
URL: https://github.com/apache/kafka/pull/15836#issuecomment-2096392283
Resolved conflict with `trunk`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
gaurav-narula commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1590274476
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -787,9 +803,37 @@ class FetchSessionCache(private val maxEntries: Int,
}
}
}
+object
chia7712 commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1590078150
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -690,6 +702,10 @@ class FetchSessionCache(private val maxEntries: Int,
* 2. B is considered "stale"
gaurav-narula commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1589956639
##
core/src/main/scala/kafka/server/BrokerServer.scala:
##
@@ -389,9 +389,17 @@ class BrokerServer(
authorizer = config.createNewAuthorizer()
chia7712 commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1589821796
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -430,7 +438,10 @@ class FullFetchContext(private val time: Time,
}
cachedPartitions
}
gaurav-narula commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1589418655
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -583,9 +595,13 @@ case class EvictableKey(privileged: Boolean, size: Int,
id: Int) extends Compara
AndrewJSchofield commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1589348685
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -395,19 +396,27 @@ object FullFetchContext {
* The fetch context for a full fetch request.
gaurav-narula commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1589339685
##
core/src/main/scala/kafka/server/KafkaServer.scala:
##
@@ -541,9 +541,17 @@ class KafkaServer(
}.toMap
}
-val fetchManager =
gaurav-narula commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1589305094
##
core/src/main/scala/kafka/server/BrokerServer.scala:
##
@@ -389,9 +389,17 @@ class BrokerServer(
authorizer = config.createNewAuthorizer()
OmniaGM commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586403158
##
core/src/main/scala/kafka/server/BrokerServer.scala:
##
@@ -389,9 +389,17 @@ class BrokerServer(
authorizer = config.createNewAuthorizer()
soarez commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586395026
##
core/src/main/scala/kafka/server/BrokerServer.scala:
##
@@ -389,9 +389,17 @@ class BrokerServer(
authorizer = config.createNewAuthorizer()
gaurav-narula commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586365674
##
core/src/main/scala/kafka/server/BrokerServer.scala:
##
@@ -389,9 +389,17 @@ class BrokerServer(
authorizer = config.createNewAuthorizer()
gaurav-narula commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586362012
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -603,14 +619,16 @@ class FetchSessionCache(private val maxEntries: Int,
// A map containing
OmniaGM commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586165970
##
core/src/test/scala/unit/kafka/server/FetchSessionTest.scala:
##
@@ -1932,6 +1932,26 @@ class FetchSessionTest {
}
assertEquals(partitions,
OmniaGM commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586165271
##
core/src/main/scala/kafka/server/KafkaServer.scala:
##
@@ -541,9 +541,17 @@ class KafkaServer(
}.toMap
}
-val fetchManager = new
OmniaGM commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586164282
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -789,7 +807,15 @@ class FetchSessionCache(private val maxEntries: Int,
}
class FetchManager(private
OmniaGM commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586116968
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -430,6 +439,9 @@ class FullFetchContext(private val time: Time,
}
cachedPartitions
}
+
OmniaGM commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586109183
##
core/src/main/scala/kafka/server/BrokerServer.scala:
##
@@ -389,9 +389,17 @@ class BrokerServer(
authorizer = config.createNewAuthorizer()
OmniaGM commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1586084377
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -603,14 +619,16 @@ class FetchSessionCache(private val maxEntries: Int,
// A map containing sessions
chia7712 commented on code in PR #15836:
URL: https://github.com/apache/kafka/pull/15836#discussion_r1585489570
##
core/src/main/scala/kafka/server/FetchSession.scala:
##
@@ -603,14 +619,16 @@ class FetchSessionCache(private val maxEntries: Int,
// A map containing sessions
gaurav-narula commented on PR #15836:
URL: https://github.com/apache/kafka/pull/15836#issuecomment-2086478984
The following images show lock profiles collected using async-profiler
before and after this change with numCacheShards = numIoThreads = 64 and
demonstrates significant reduction
gaurav-narula opened a new pull request, #15836:
URL: https://github.com/apache/kafka/pull/15836
KIP-227 introduced in-memory caching of FetchSessions. Brokers with a large
number of Fetch requests suffer from contention on trying to acquire a lock on
FetchSessionCache.
This change
33 matches
Mail list logo
