[jira] [Updated] (DRILL-5602) Repeated List Vector fails to initialize the offset vector
[
https://issues.apache.org/jira/browse/DRILL-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5602:
---
Summary: Repeated List Vector fails to initialize the offset vector (was:
Vector corruption when allocating a repeated, variable-width vector)
> Repeated List Vector fails to initialize the offset vector
> --
>
> Key: DRILL-5602
> URL: https://issues.apache.org/jira/browse/DRILL-5602
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> The query in DRILL-5513 highlighted a problem described in DRILL-5594: that
> the external sort did not properly allocate its spill batch vectors, and
> instead allowed them to grow by doubling. While fixing that issue, a new
> issue became clear.
> The method to allocate a repeated map vector, however, has a serious bug, as
> described in DRILL-5530: value vectors do not zero-fill the first allocation
> for a vector (though subsequent reallocs are zero-filled.)
> If the code worked correctly, here is the behavior when writing to the first
> element of the list:
> * Access the offset vector at offset 0. Should be 0.
> * Write the new value at that offset. Since the first offset is 0, the first
> value is written at 0 in the value vector.
> * Write into offset 1 the value at offset 0 plus the length of the new value.
> But, the offset vector is not initialized to zero. Instead, offset 0 contains
> the value 16 million. Now:
> * Access the offset vector at offset 0. Value is 16 million.
> * Write the new value at that offset. Write at position 16 million. This
> requires growing the value vector from its present size to 16 MB.
> The problem is here in {{RepeatedMapVector}}:
> {code}
> public void allocateOffsetsNew(int groupCount) {
> offsets.allocateNew(groupCount + 1);
> }
> {code}
> Notice that there is no code to set the value at offset 0.
> Then, in the {{UInt4Vector}}:
> {code}
> public void allocateNew(final int valueCount) {
> allocateBytes(valueCount * 4);
> }
> private void allocateBytes(final long size) {
> ...
> data = allocator.buffer(curSize);
> ...
> {code}
> The above eventually calls the Netty memory allocator, which explicitly
> states that, for performance reasons, it does not zero-fill its buffers.
> The code works in small tests because the new buffer comes from Java direct
> memory, which *does* zero-fill the buffer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5602) Repeated List Vector fails to initialize the offset vector
[
https://issues.apache.org/jira/browse/DRILL-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5602:
---
Description:
The code that allocates a new {{RepeatedListVector}} does not initialize the
first offset to zero as required:
{code}
@Override
public void allocateNew(int valueCount, int innerValueCount) {
clear();
getOffsetVector().allocateNew(valueCount + 1);
getMutator().reset();
}
{code}
Since Netty does not zero-fill vectors, the result is vector corruption.
If the code worked correctly, here is the behavior when writing to the first
element of the list:
* Access the offset vector at offset 0. Should be 0.
* Write the new value at that offset. Since the first offset is 0, the first
value is written at 0 in the value vector.
* Write into offset 1 the value at offset 0 plus the length of the new value.
But, the offset vector is not initialized to zero. Instead, offset 0 contains
the value 16 million. Now:
* Access the offset vector at offset 0. Value is 16 million.
* Write the new value at that offset. Write at position 16 million. This
requires growing the value vector from its present size to 16 MB.
was:
The query in DRILL-5513 highlighted a problem described in DRILL-5594: that the
external sort did not properly allocate its spill batch vectors, and instead
allowed them to grow by doubling. While fixing that issue, a new issue became
clear.
The method to allocate a repeated map vector, however, has a serious bug, as
described in DRILL-5530: value vectors do not zero-fill the first allocation
for a vector (though subsequent reallocs are zero-filled.)
If the code worked correctly, here is the behavior when writing to the first
element of the list:
* Access the offset vector at offset 0. Should be 0.
* Write the new value at that offset. Since the first offset is 0, the first
value is written at 0 in the value vector.
* Write into offset 1 the value at offset 0 plus the length of the new value.
But, the offset vector is not initialized to zero. Instead, offset 0 contains
the value 16 million. Now:
* Access the offset vector at offset 0. Value is 16 million.
* Write the new value at that offset. Write at position 16 million. This
requires growing the value vector from its present size to 16 MB.
The problem is here in {{RepeatedMapVector}}:
{code}
public void allocateOffsetsNew(int groupCount) {
offsets.allocateNew(groupCount + 1);
}
{code}
Notice that there is no code to set the value at offset 0.
Then, in the {{UInt4Vector}}:
{code}
public void allocateNew(final int valueCount) {
allocateBytes(valueCount * 4);
}
private void allocateBytes(final long size) {
...
data = allocator.buffer(curSize);
...
{code}
The above eventually calls the Netty memory allocator, which explicitly states
that, for performance reasons, it does not zero-fill its buffers.
The code works in small tests because the new buffer comes from Java direct
memory, which *does* zero-fill the buffer.
> Repeated List Vector fails to initialize the offset vector
> --
>
> Key: DRILL-5602
> URL: https://issues.apache.org/jira/browse/DRILL-5602
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> The code that allocates a new {{RepeatedListVector}} does not initialize the
> first offset to zero as required:
> {code}
> @Override
> public void allocateNew(int valueCount, int innerValueCount) {
> clear();
> getOffsetVector().allocateNew(valueCount + 1);
> getMutator().reset();
> }
> {code}
> Since Netty does not zero-fill vectors, the result is vector corruption.
> If the code worked correctly, here is the behavior when writing to the first
> element of the list:
> * Access the offset vector at offset 0. Should be 0.
> * Write the new value at that offset. Since the first offset is 0, the first
> value is written at 0 in the value vector.
> * Write into offset 1 the value at offset 0 plus the length of the new value.
> But, the offset vector is not initialized to zero. Instead, offset 0 contains
> the value 16 million. Now:
> * Access the offset vector at offset 0. Value is 16 million.
> * Write the new value at that offset. Write at position 16 million. This
> requires growing the value vector from its present size to 16 MB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5602) Repeated List Vector fails to initialize the offset vector
[
https://issues.apache.org/jira/browse/DRILL-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5602:
---
Issue Type: Sub-task (was: Bug)
Parent: DRILL-5601
> Repeated List Vector fails to initialize the offset vector
> --
>
> Key: DRILL-5602
> URL: https://issues.apache.org/jira/browse/DRILL-5602
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> The code that allocates a new {{RepeatedListVector}} does not initialize the
> first offset to zero as required:
> {code}
> @Override
> public void allocateNew(int valueCount, int innerValueCount) {
> clear();
> getOffsetVector().allocateNew(valueCount + 1);
> getMutator().reset();
> }
> {code}
> Since Netty does not zero-fill vectors, the result is vector corruption.
> If the code worked correctly, here is the behavior when writing to the first
> element of the list:
> * Access the offset vector at offset 0. Should be 0.
> * Write the new value at that offset. Since the first offset is 0, the first
> value is written at 0 in the value vector.
> * Write into offset 1 the value at offset 0 plus the length of the new value.
> But, the offset vector is not initialized to zero. Instead, offset 0 contains
> the value 16 million. Now:
> * Access the offset vector at offset 0. Value is 16 million.
> * Write the new value at that offset. Write at position 16 million. This
> requires growing the value vector from its present size to 16 MB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5602) Repeated List Vector fails to initialize the offset vector
[
https://issues.apache.org/jira/browse/DRILL-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5602:
---
Issue Type: Bug (was: Sub-task)
Parent: (was: DRILL-5601)
> Repeated List Vector fails to initialize the offset vector
> --
>
> Key: DRILL-5602
> URL: https://issues.apache.org/jira/browse/DRILL-5602
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> The code that allocates a new {{RepeatedListVector}} does not initialize the
> first offset to zero as required:
> {code}
> @Override
> public void allocateNew(int valueCount, int innerValueCount) {
> clear();
> getOffsetVector().allocateNew(valueCount + 1);
> getMutator().reset();
> }
> {code}
> Since Netty does not zero-fill vectors, the result is vector corruption.
> If the code worked correctly, here is the behavior when writing to the first
> element of the list:
> * Access the offset vector at offset 0. Should be 0.
> * Write the new value at that offset. Since the first offset is 0, the first
> value is written at 0 in the value vector.
> * Write into offset 1 the value at offset 0 plus the length of the new value.
> But, the offset vector is not initialized to zero. Instead, offset 0 contains
> the value 16 million. Now:
> * Access the offset vector at offset 0. Value is 16 million.
> * Write the new value at that offset. Write at position 16 million. This
> requires growing the value vector from its present size to 16 MB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
