billoley opened a new issue, #2789:
URL: https://github.com/apache/accumulo/issues/2789
Running version 1.9.3, observed an exception in the tserver logs from Tablet
(now in TabletBase) --> "Underlying iterator yielded to a position that does
not follow the last key returned: " The exception didn't make sense because
the last key returned (results.get(results.size() - 1)) was not in the seek
range. While writing a unit test to reproduce the issue, three separate issues
in LookupTask were discovered.
1)
In LookupTask.run(), session.queries is a HashMap (therefore not sorted), so
when List<KVEntry> results = new ArrayList<>(); which contains the accumulated
results for all subsequent KeyExtents gets passed to successive tablets, a
yield could easily happen at a key that is before the last entry in results.
("Underlying iterator yielded to a position that does not follow the last key
returned: ") LookupTask.run() should pass an empty List of results and add the
returned value to the accumulated results.
2)
LookupTask.run() uses an iterator (Iterator<Entry<KeyExtent,List<Range>>>
iter = session.queries.entrySet().iterator()). If there are unfinished ranges
on a tablet that is not closed, it calls session.queries.put(entry.getKey(),
lookupResult.unfinishedRanges) which is modifying the container that is being
iterated over inside iteration loop. This results in a
ConcurrentModificationException the next time iter.next() is called.
3)
After #1 and #2 are corrected and the unfinished ranges are re-added to
session.queries for a subsequent call, there are issues with
partScan/partNextKey/partNextKeyInclusive
a) These variables are not used until after the loop exits, so they will be
overwritten any time there is a lookupResult with unfinishedRanges on a tablet
that is not closed.
b) Once we complete scanning the extent/ranges that were previously
unfinished, the last one encoutered (is this desired) will then be in
partScan/partNextKey/partNextKeyInclusive and fullscans. Both are returned in
the thrift response..... and this causes a NullPointerException in
TabletServerBatchReaderIterator at line 554 in trackScanning()
ListIterator<Range> iterator = unscanned.get(ke).listIterator()
sequence of events:
Map<KeyExtent,List<Range>> unscanned starts with all requested values
remove all failures from unscanned
remove all fullScans from unscanned
then....
if (scanResult.partScan != null) {
KeyExtent ke = new KeyExtent(scanResult.partScan);
Key nextKey = new Key(scanResult.partNextKey);
ListIterator<Range> iterator = unscanned.get(ke).listIterator();
NullPointerException because the scan on ke was completed and it is in
fullScans, thus removed from the unscanned container
Encountered in version 1.9.3, but these issues exist in much older versions
up to the current.
Reproduced by adding another test to YieldScannersIt that added splits to
the table prior to adding data, ensuring that LookupTask received a request
that included more than one KeyExtent -> List<Range> entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
