[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-3976: - Release Note: KeyValueBlockIterator is not skipping valid blocks. Resolution: Not A Problem Status: Resolved (was: Patch Available) > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > Labels: pull-request-available > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > Additionally, the current implementation of KeyValueBlockIterator#seekLast > depends on the internal RocksDB iterators seekLast() method, which will skip > to the last key in the DB regardless of whether it matches the filter or not. > This could be different from last key according to the filter. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3976: Status: Patch Available (was: Open) > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > Labels: pull-request-available > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > Additionally, the current implementation of KeyValueBlockIterator#seekLast > depends on the internal RocksDB iterators seekLast() method, which will skip > to the last key in the DB regardless of whether it matches the filter or not. > This could be different from last key according to the filter. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3976: - Labels: pull-request-available (was: ) > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > Labels: pull-request-available > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > Additionally, the current implementation of KeyValueBlockIterator#seekLast > depends on the internal RocksDB iterators seekLast() method, which will skip > to the last key in the DB regardless of whether it matches the filter or not. > This could be different from last key according to the filter. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Rose updated HDDS-3976: - Description: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. Additionally, the current implementation of KeyValueBlockIterator#seekLast depends on the internal RocksDB iterators seekLast() method, which will skip to the last key in the DB regardless of whether it matches the filter or not. This could be different from last key according to the filter. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. was: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > Additionally, the current implementation of KeyValueBlockIterator#seekLast > depends on the internal RocksDB iterators seekLast() method, which will skip > to the last key in the DB regardless of whether it matches the filter or not. > This could be different from last key according to the filter. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Rose updated HDDS-3976: - Description: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. was: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. Additionally, the current implementation modifies the internal iterator's state on hasNext() calls, so multiple consecutive hasNext() calls will actually skip over elements. While this is not necessarily a bug, it is atypical iterator behavior and should be resolved to prevent user error. > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Rose updated HDDS-3976: - Description: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. Additionally, the current implementation modifies the internal iterator's state on hasNext() calls, so multiple consecutive hasNext() calls will actually skip over elements. While this is not necessarily a bug, it is atypical iterator behavior and should be resolved to prevent user error. was: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock() etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. Additionally, the current implementation modifies the internal iterator's state on hasNext() calls, so multiple consecutive hasNext() calls will actually skip over elements. While this is not necessarily a bug, it is atypical iterator behavior and should be resolved to prevent user error. > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. > Additionally, the current implementation modifies the internal iterator's > state on hasNext() calls, so multiple consecutive hasNext() calls will > actually skip over elements. While this is not necessarily a bug, it is > atypical iterator behavior and should be resolved to prevent user error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
