[
https://issues.apache.org/jira/browse/DRILL-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005685#comment-16005685
]
Paul Rogers edited comment on DRILL-5504 at 5/11/17 1:31 AM:
-------------------------------------------------------------
Mini-design:
1. Create a BatchValidator class to iterate over vectors in a batch and
validate each.
2. Add a new session option, debug.validate_vectors to enable validation.
3. Modify IteratorValidatorBatchIterator to invoke the batch validator on each
batch, if the above option is set.
4. Add another new session option, debug.validate_iterators, to enable the
iterator validator.
5. Modify ImplCreator to consider both assertions enabled OR the
debug.validate_iterator option to decide to inject the iterator validator.
The result will be that a production, non-debug user can enable validation for
just a problematic query, without having to restart the Drillbit with
assertions enabled.
Much of the validation logic requires knowledge of the specific vector type.
Nullable vectors place the is-set vector in the generated subclasses rather
than in the common base class. And so on. Validation would thus be done by
either adjusting the generated vectors to make the common parts available, or
pushing validation into the generated vectors.
As a first step, however, the code will be in a single class to allow use by a
particular user for a specific problem. Once that is resolved, a second version
can make broader changes.
was (Author: paul-rogers):
Mini-design:
1. Create a BatchValidator class to iterate over vectors in a batch and
validate each.
2. Create a VectorValidator class to validate each kind of vector that lends
itself to validation.
3. Add a new session option, debug.validate_vectors to enable validation.
4. Modify IteratorValidatorBatchIterator to invoke the batch validator on each
batch, if the above option is set.
5. Add another new session option, debug.validate_iterators, to enable the
iterator validator.
6. Modify ImplCreator to consider both assertions enabled OR the
debug.validate_iterator option to decide to inject the iterator validator.
The result will be that a production, non-debug user can enable validation for
just a problematic query, without having to restart the Drillbit with
assertions enabled.
> Vector validator to diagnose offset vector issues
> -------------------------------------------------
>
> Key: DRILL-5504
> URL: https://issues.apache.org/jira/browse/DRILL-5504
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.10.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Priority: Minor
> Fix For: 1.11.0
>
>
> DRILL-5470 describes a case in which an offset vector appears to have become
> corrupted, yielding a bogus field-length value that is orders of magnitude
> larger than the vector that contains the data.
> Debugging such cases is slow and tedious. To help, we propose to create a
> "vector validator" that spins through vectors looking for problems.
> Then, to allow the validator to be used in the field, extend the "iterator
> validator batch iterator" to optionally allow vector validation on each batch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
