Robert Purdom created ARROW-17169:
-------------------------------------
Summary: [Go] goPanicIndex in firstTimeBitmapWriter.Finish()
Key: ARROW-17169
URL: https://issues.apache.org/jira/browse/ARROW-17169
Project: Apache Arrow
Issue Type: Bug
Components: Go
Affects Versions: 8.0.1, 9.0.0
Environment: go (1.18.3), Linux, AMD64
Reporter: Robert Purdom
I'm working with complex parquet files with 500+ "root" columns where some
fields are lists of structs, internally referred to as 'topics'. Some of these
structs have 100's of columns. When reading a particular topic, I get an Index
Panic at the line indicated below. This error occurs when the value for the
topic is Null, as in, for this particular root record, this topic has no data.
The root is household data, the topic is auto, so the error occurs when the
household has no autos. The auto field is a Nullable List of Struct.
{code:go}
/* Finish() was called from defLevelsToBitmapInternal.
data values when panic occurs....
bw.length == 17531
bw.bitMask == 1
bw.pos == 3424
bw.length == 17531
len(bw.Buf) == 428
cap(bw.Buf) == 448
bw.byteOffset == 428
bw.curByte == 0
*/
// bitmap_writer.go
func (bw *firstTimeBitmapWriter) Finish() {
// store curByte into the bitmap
if bw.length >0&& bw.bitMask !=0x01|| bw.pos < bw.length {
bw.buf[int(bw.byteOffset)] = bw.curByte // <---- Panic index
}
}
{code}
In every case, when the panic occurs, bw.byteOffset == len(bw.Buf). I tested
the below modification and it does remedy the bug. However, it's probably only
masking the actual bug.
{code:go}
// Test version: No Panic
func (bw *firstTimeBitmapWriter) Finish() {
// store curByte into the bitmap
if bw.length > 0 && bw.bitMask != 0x01 || bw.pos < bw.length {
if bw.byteOffset == len(bw.Buf) {
bw.buf = append(bw.buf, bw.curByte)
} else {
bw.buf[int(bw.byteOffset)] = bw.curByte
}
}
}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
