[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r471459574
##
File path:
streams/src/test/java/org/apache/kafka/streams/state/internals/AbstractKeyValueStoreTest.java
##
@@ -188,7 +188,55 @@ public void testPutGetRange() {
}
@Test
-public void testPutGetRangeWithDefaultSerdes() {
+public void testPutGetReverseRange() {
Review comment:
@ableegoldman I have extended `RocksDBTimestampedStoreTest` to use
`reverseAll` and `reverseRange` as part of the current tests.
Unfortunately, `AbstractKeyValueStoreTest` tests do not fit with the
creation path of Timestamped stores as pre inserted data is required.
Will add this to the same JIRA ticket to consider when refactoring iterators
and tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r471451498
##
File path:
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStore.java
##
@@ -374,7 +412,7 @@ public Bytes peekNextKey() {
if (next == null) {
return allDone();
} else {
-if (comparator.compare(next.key.get(), upperBoundKey) <= 0) {
+if (comparator.compare(next.key.get(), lastKey) <= 0) {
Review comment:
https://issues.apache.org/jira/browse/KAFKA-10409
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r471449807
##
File path:
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStore.java
##
@@ -374,7 +412,7 @@ public Bytes peekNextKey() {
if (next == null) {
return allDone();
} else {
-if (comparator.compare(next.key.get(), upperBoundKey) <= 0) {
+if (comparator.compare(next.key.get(), lastKey) <= 0) {
Review comment:
Agree. I will continue the current approoach and create an issue to
follow up this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r470536034
##
File path:
streams/src/test/java/org/apache/kafka/streams/state/internals/AbstractKeyValueStoreTest.java
##
@@ -188,7 +188,55 @@ public void testPutGetRange() {
}
@Test
-public void testPutGetRangeWithDefaultSerdes() {
+public void testPutGetReverseRange() {
Review comment:
Just realized that, I also thought that path was tested. Good catch!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r469906366
##
File path:
streams/src/test/java/org/apache/kafka/streams/state/internals/CompositeReadOnlyKeyValueStoreTest.java
##
@@ -199,11 +236,44 @@ public void shouldSupportRangeAcrossMultipleKVStores() {
cache.put("x", "x");
final List> results =
toList(theStore.range("a", "e"));
+assertArrayEquals(
+asList(
+new KeyValue<>("a", "a"),
+new KeyValue<>("b", "b"),
+new KeyValue<>("c", "c"),
+new KeyValue<>("d", "d")
+).toArray(),
+results.toArray());
+}
+
+@Test
+public void shouldSupportReverseRangeAcrossMultipleKVStores() {
+final KeyValueStore cache = newStoreInstance();
+stubProviderTwo.addStore(storeName, cache);
+
+stubOneUnderlying.put("a", "a");
+stubOneUnderlying.put("b", "b");
+stubOneUnderlying.put("z", "z");
+
+cache.put("c", "c");
+cache.put("d", "d");
+cache.put("x", "x");
+
+final List> results =
toList(theStore.reverseRange("a", "e"));
assertTrue(results.contains(new KeyValue<>("a", "a")));
assertTrue(results.contains(new KeyValue<>("b", "b")));
assertTrue(results.contains(new KeyValue<>("c", "c")));
assertTrue(results.contains(new KeyValue<>("d", "d")));
assertEquals(4, results.size());
+//FIXME: order does not hold between stores, how to validate order
here?
Review comment:
ack. make sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r469315828
##
File path:
streams/src/test/java/org/apache/kafka/streams/state/internals/CachingKeyValueStoreTest.java
##
@@ -339,12 +366,24 @@ public void
shouldThrowIfTryingToDoRangeQueryOnClosedCachingStore() {
store.range(bytesKey("a"), bytesKey("b"));
}
+@Test(expected = InvalidStateStoreException.class)
Review comment:
migrated the whole class. will apply this to the other PRs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r469315300
##
File path:
streams/src/test/java/org/apache/kafka/streams/state/internals/CachingKeyValueStoreTest.java
##
@@ -294,13 +305,27 @@ public void shouldIterateOverRange() {
assertEquals(items, results.size());
}
+@Test
+public void shouldReverseIterateOverRange() {
+final int items = addItemsToCache();
+final KeyValueIterator range =
+store.reverseRange(bytesKey(String.valueOf(0)),
bytesKey(String.valueOf(items)));
+final List results = new ArrayList<>();
+while (range.hasNext()) {
+results.add(range.next().key);
+}
+assertEquals(items, results.size());
Review comment:
ack. added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r469314938
##
File path:
streams/src/test/java/org/apache/kafka/streams/state/internals/AbstractKeyValueStoreTest.java
##
@@ -422,6 +503,21 @@ public void
shouldNotThrowInvalidRangeExceptionWithNegativeFromKey() {
" Note that the built-in numerical serdes do not follow
this for negative numbers")
);
}
+}
+
+@Test
+public void
shouldNotThrowInvalidReverseRangeExceptionWithNegativeFromKey() {
Review comment:
ack. added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #9137: KAFKA-9929: Support reverse iterator on KeyValueStore
jeqo commented on a change in pull request #9137:
URL: https://github.com/apache/kafka/pull/9137#discussion_r469225368
##
File path:
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBRangeIterator.java
##
@@ -29,32 +29,41 @@
// comparator to be pluggable, and the default is lexicographic, so it's
// safe to just force lexicographic comparator here for now.
private final Comparator comparator =
Bytes.BYTES_LEXICO_COMPARATOR;
-private final byte[] rawToKey;
+private final byte[] rawLastKey;
+private final boolean reverse;
RocksDBRangeIterator(final String storeName,
final RocksIterator iter,
final Set>
openIterators,
final Bytes from,
- final Bytes to) {
-super(storeName, iter, openIterators);
-iter.seek(from.get());
-rawToKey = to.get();
-if (rawToKey == null) {
+ final Bytes to,
+ final boolean reverse) {
+super(storeName, iter, openIterators, reverse);
+this.reverse = reverse;
+if (reverse) {
+iter.seekForPrev(to.get());
+rawLastKey = from.get();
+} else {
+iter.seek(from.get());
+rawLastKey = to.get();
+}
+if (rawLastKey == null) {
throw new NullPointerException("RocksDBRangeIterator: RawToKey is
null for key " + to);
Review comment:
right. I've moved this into each condition to set a correct exception
message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
