joechenrh commented on code in PR #880:
URL: https://github.com/apache/arrow-go/pull/880#discussion_r3567705721
##########
parquet/file/page_reader.go:
##########
@@ -91,6 +99,43 @@ func (p *page) Data() []byte { return
p.buf.Bytes() }
func (p *page) NumValues() int32 { return p.nvals }
func (p *page) Encoding() format.Encoding { return p.encoding }
+// dataPageBase holds the members shared by DataPageV1/V2 but not dictionary
+// pages: the rep/def level buffer accessor and, in streaming mode, the
+// incremental value source.
+type dataPageBase struct {
+ page
+
+ // valueBuf is only set for streaming mode to read values incrementally.
+ // In such mode, buf only holds the rep+def level region.
+ valueBuf streaming.ValueBuffer
+ // clip bounds a read's batch to keep one Decode's aliased values near
the buffer
+ // size; 0 for non-streaming pages.
+ clip int64
+}
+
+func (d *dataPageBase) valueBuffer() streaming.ValueBuffer { return d.valueBuf
}
+func (d *dataPageBase) levelBuffer() []byte { return
d.buf.Bytes() }
+
+func (d *dataPageBase) valueClip() int64 { return d.clip }
+
+func (d *dataPageBase) setStreamingValues(mem memory.Allocator,
lenUncompressed int, levelBuf []byte, valReader, rawSrc io.Reader, closer
io.Closer) {
+ d.buf = memory.NewBufferBytes(levelBuf)
Review Comment:
Small follow-up on this one: the first cut in `7529291d` actually tripped
CI. Coupling `readLevelData` to `p.mem` broke two internal tests
(`TestReadLevelDataRLELengthBound` / `TestReadLevelDataBitPacked`) that spin up
a bare `serializedPageReader` with no allocator, so `buf.Resize` hit a
nil-deref.
Fixed in
https://github.com/apache/arrow-go/commit/5f00dc754d9af6e66924b1d92c38eff056e869fa
— the readers now get a `memory.DefaultAllocator` and assert on the
`*memory.Buffer` return.
I'd been running a filtered `-run` locally that skipped those two, which is
how I missed them. Re-ran the actual CI recipe this time — `go test -race -tags
assert ./parquet/...` plus the `,noasm` variant with the test-data env set —
and it's fully green now. 🟢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
