Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/518
The open question was how we can discard a partly-built record during
recovery. As far as I can tell (veterans, please correct me), the
JSONRecordReader keeps track of the record count. So, all we have to do is not
increment the count when we want to discard a record. Look in
JSONRecordReader.next( )
...
outside: while(recordCount < DEFAULT_ROWS_PER_BATCH) {
writer.setPosition(recordCount); // Sets the position for the next
read.
write = jsonReader.write(writer); // Write the record. We can catch
errors
// and recover here??
...
recordCount++; // Don't do this on a bad record
...
writer.setValueCount(recordCount); // The record reader controls the
record count.
This seems to show the elements of a solution:
1. Try to read the record.
2. If a failure occurs, catch it here and clean up, as in the previous post.
3. Don't increment the record count. We reuse the current one on the next
record read.
Now the only open question is how we clean up the in-flight record in case
some columns are not present in the next record. Anyone know how to set a
vector position to null (for optional) default value (for required) or
zero-length (for repeated)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
