[
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525212#comment-15525212
]
ASF GitHub Bot commented on DRILL-4653:
---------------------------------------
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/518
The open question was how we can discard a partly-built record during
recovery. As far as I can tell (veterans, please correct me), the
JSONRecordReader keeps track of the record count. So, all we have to do is not
increment the count when we want to discard a record. Look in
JSONRecordReader.next( )
...
outside: while(recordCount < DEFAULT_ROWS_PER_BATCH) {
writer.setPosition(recordCount); // Sets the position for the next
read.
write = jsonReader.write(writer); // Write the record. We can catch
errors
// and recover here??
...
recordCount++; // Don't do this on a bad record
...
writer.setValueCount(recordCount); // The record reader controls the
record count.
This seems to show the elements of a solution:
1. Try to read the record.
2. If a failure occurs, catch it here and clean up, as in the previous post.
3. Don't increment the record count. We reuse the current one on the next
record read.
Now the only open question is how we clean up the in-flight record in case
some columns are not present in the next record. Anyone know how to set a
vector position to null (for optional) default value (for required) or
zero-length (for repeated)?
> Malformed JSON should not stop the entire query from progressing
> ----------------------------------------------------------------
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - JSON
> Affects Versions: 1.6.0
> Reporter: subbu srinivasan
> Fix For: Future
>
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something
> similar to a setting of (ignore.malformed.json) would help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
