aliehsaeedii commented on code in PR #23184:
URL: https://github.com/apache/kafka/pull/23184#discussion_r3805410739
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/StoreQueryUtils.java:
##########
@@ -128,29 +128,33 @@ public static <R> QueryResult<R> handleBasicQueries(
final QueryResult<R> result;
final QueryHandler<?> handler =
QUERY_HANDLER_MAP.get(query.getClass());
- synchronized (position) {
- if (handler == null) {
- result = QueryResult.forUnknownQueryType(query, store);
- } else if (context == null || !isPermitted(position,
positionBound, context.taskId().partition())) {
- result = QueryResult.notUpToBound(
- position,
- positionBound,
- context == null ? null : context.taskId().partition()
- );
- } else {
- result = ((QueryHandler<R>) handler).apply(
- query,
- positionBound,
- config,
- store
- );
- }
- if (config.isCollectExecutionInfo()) {
- result.addExecutionInfo(
- "Handled in " + store.getClass() + " in " +
(System.nanoTime() - start) + "ns"
- );
+ // Lock order must match the write paths (store monitor, then
position),
+ // otherwise concurrent put/query can deadlock (KAFKA-19629).
+ synchronized (store) {
Review Comment:
Locking store→position here introduces a new deadlock for
`MemoryNavigableLRUCache`: its restore callback in `MemoryLRUCache.init` holds
`position` and then calls the `synchronized` `put()`, i.e. position→store — the
reverse of this order. An IQ query can now deadlock with changelog restore
(including on standby stores). `MemoryLRUCache`'s restore path needs the same
store→position order (or should call a non-`synchronized` internal put, like
`InMemoryKeyValueStore.putInternal` does).
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryWindowStore.java:
##########
@@ -488,25 +488,29 @@ public <R> QueryResult<R> query(final Query<R> query,
final PositionBound positionBound,
final QueryConfig config) {
- synchronized (position) {
- // Mirror RocksDBStore#query: under READ_UNCOMMITTED, expose the
writes staged in the
- // transaction buffer since the last commit by merging the
buffer's pending position
- // deltas into a copy of the committed position. READ_COMMITTED
(and the
- // non-transactional store) query the committed position directly.
- final Position queryPosition;
- if (transactionBuffer != null && config.getIsolationLevel() ==
IsolationLevel.READ_UNCOMMITTED) {
- queryPosition =
position.copy().merge(transactionBuffer.pendingPosition());
- } else {
- queryPosition = position;
+ // Lock order must match the write paths (store monitor, then
position),
+ // otherwise concurrent put/query can deadlock (KAFKA-19629).
+ synchronized (this) {
Review Comment:
Same as `InMemorySessionStore`: this store never locks `this` elsewhere
(`put` and reads use `position` plus concurrent maps), so this outer
`synchronized(this)` guards nothing, and the comment's "store monitor, then
position" doesn't match `put`, which locks only `position`. Drop it, or fix the
comment.
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/InMemorySessionStore.java:
##########
@@ -476,25 +476,29 @@ public <R> QueryResult<R> query(final Query<R> query,
final PositionBound positionBound,
final QueryConfig config) {
- synchronized (position) {
- // Mirror RocksDBStore#query: under READ_UNCOMMITTED, expose the
writes staged in the
- // transaction buffer since the last commit by merging the
buffer's pending position
- // deltas into a copy of the committed position. READ_COMMITTED
(and the
- // non-transactional store) query the committed position directly.
- final Position queryPosition;
- if (transactionBuffer != null && config.getIsolationLevel() ==
IsolationLevel.READ_UNCOMMITTED) {
- queryPosition =
position.copy().merge(transactionBuffer.pendingPosition());
- } else {
- queryPosition = position;
+ // Lock order must match the write paths (store monitor, then
position),
+ // otherwise concurrent put/query can deadlock (KAFKA-19629).
+ synchronized (this) {
Review Comment:
This store never locks `this` anywhere else — `put` and the reads use
`position` and concurrent maps — so this outer `synchronized(this)` orders
against nothing, and there was no store/position inversion here to fix. The
comment is also off: `put` takes only the position lock, not the store monitor.
Drop the `synchronized(this)`, or fix the comment.
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java:
##########
@@ -536,21 +536,25 @@ public <R> QueryResult<R> query(
final PositionBound positionBound,
final QueryConfig config) {
- synchronized (position) {
- final Position queryPosition;
- if (config.getIsolationLevel() == IsolationLevel.READ_COMMITTED) {
- queryPosition = position;
- } else {
- queryPosition =
position.copy().merge(dbAccessor.uncommittedPositionDeltas());
+ // Lock order must match the write paths (store monitor, then
position),
+ // otherwise concurrent put/query can deadlock (KAFKA-19629).
+ synchronized (this) {
Review Comment:
No tests added for a concurrency/deadlock fix. Can we add a regression test
that runs concurrent `put` and `range` (IQ) and asserts no deadlock within a
timeout? KAFKA-19629 has a concrete repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
