equanz commented on PR #20179: URL: https://github.com/apache/pulsar/pull/20179#issuecomment-1539314184
@poorbarcode * [Q-1] Yes. * [Q-2] When I created this PR, I introduced Map structure because it was necessary. For more details, please see the description("2. Keep the last sent position per key."). I just noticed we could avoid introducing a Map structure if we implemented the last sent position feature, like the mark delete position and the individually deleted messages feature. In other words, - The position is already scheduled to be sent. - All positions less than or equal to it are already scheduled to be sent. - Manage individually sent positions to update the position as expected. Consider the following flow. 1. Assume that the [entries](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L174) has the following messages, - `msg-1`, key: `key-a`, position: `1:1` - `msg-2`, key: `key-a`, position: `1:2` - `msg-3`, key: `key-a`, position: `1:3` - `msg-4`, key: `key-b`, position: `1:4` - `msg-5`, key: `key-b`, position: `1:5` - `msg-6`, key: `key-b`, position: `1:6` the dispatcher has two consumers (`c1` `messagesForC` is 1, `c2` `messageForC` is 1000), and the selector will return `c1` if `key-a` and `c2` if `key-b`. 2. Send `msg-1` to `c1` and `msg-4` - `msg-6` to `c2`. - So, the current last sent position is `1:1` and the individually sent positions is `[[1:3, 1:6]]` (list of closed intervals. not list of list). - `c1` never acknowledge `msg-1`. **scenario A** <ol start="3"> <li>Add new consumer <code class="notranslate">c3</code>, and the selector will return <code class="notranslate">c3</code> if <code class="notranslate">key-a</code>.</li> <li>Can't send <code class="notranslate">msg-2</code> - <code class="notranslate">msg-3</code> to <code class="notranslate">c3</code> because <code class="notranslate">1:2</code>, and <code class="notranslate">1:3</code> are greater than the last sent position, <code class="notranslate">1:1</code>.</li> <li>Disconnect <code class="notranslate">c1</code>.</li> <li>Send <code class="notranslate">msg-1</code> - <code class="notranslate">msg-3</code> to <code class="notranslate">c3</code>.<br>Now, <code class="notranslate">c3</code> receives messages with expected order about <code class="notranslate">key-a</code>.</li> </ol> **scenario B** <ol start="3"> <li><code class="notranslate">c1</code> <code class="notranslate">messagesForC</code> is back to 999.</li> <li>Send <code class="notranslate">msg-2</code> - <code class="notranslate">msg-3</code> to <code class="notranslate">c1</code></li> <ul dir="auto"> <li>So, the current last sent position is <code class="notranslate">1:6</code>, and the individually sent positions is <code class="notranslate">[]</code>.</li> </ul> </ol> My motivation is not to improve Key_Shared but fix Key_Shared. Therefore, if the current approach is not accepted as a fix, I introduce it and remove the feature by Map per key. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org