zeroshade commented on code in PR #1609:
URL: https://github.com/apache/arrow-go/pull/1609#discussion_r4618064941
##########
arrow/cdata/cdata_exports.go:
##########
@@ -511,17 +512,24 @@ func (rr *cRecordReader) release() {
}
type cAsyncStreamHandler struct {
+ // mu guards producer: once release returns the producer may be freed,
+ // so it must not be called again.
+ mu sync.Mutex
producer *CArrowAsyncProducer
taskQueue chan taskState
ctx context.Context
}
-func asyncTaskQueue(ctx context.Context, schema *arrow.Schema, recordStream
chan<- RecordMessage, taskQueue <-chan taskState, producer
*CArrowAsyncProducer) {
+func asyncTaskQueue(ctx context.Context, schema *arrow.Schema, recordStream
chan<- RecordMessage, taskQueue <-chan taskState, h *cAsyncStreamHandler) {
defer close(recordStream)
for {
select {
case <-ctx.Done():
- C.goCallCancel(producer)
+ h.mu.Lock()
+ if h.producer != nil {
+ C.goCallCancel(h.producer)
+ }
+ h.mu.Unlock()
return
Review Comment:
Not from this PR, but related to the same spec section. On cancellation the
goroutine calls `cancel` and returns at once, so tasks still in `taskQueue`,
and any the producer sends afterwards for requests already made, are never
passed to `extract_data`.
The spec says a consumer that doesn't want a task's data "must still call
`extract_data` so that the producer can perform any required cleanup", passing
`NULL`. It also says the consumer must expect more `on_next_task` calls after
`cancel`. So the producer never gets to clean those tasks up.
Now that `asyncTaskExtract` handles a `NULL` `out`, the consumer side could
drain the queue after `cancel` and call `extract_data` with `NULL` for each
task until `release` closes the channel. That seems better as a follow-up than
in this PR. I found this by reading the code and haven't run a conforming
producer against it.
##########
arrow/cdata/cdata_exports.go:
##########
@@ -535,7 +543,11 @@ func asyncTaskQueue(ctx context.Context, schema
*arrow.Schema, recordStream chan
}
// request another batch now that we've processed this
one
- C.goCallRequest(producer, C.int64_t(1))
+ h.mu.Lock()
Review Comment:
A correction to the description. It says "the spec asks producers not to
call `release` from inside `request` while not cancelled". I can't find that in
the spec.
- The `ArrowAsyncProducer.request` docs only forbid `request` from
synchronously calling `on_next_task` or `on_error`.
- `cancel` must not call any consumer handler other than `on_error`, so
`release` is ruled out from inside `cancel`, but not from inside `request`.
That matters because `h.mu` is held across this call. If a producer called
`release` synchronously from inside `request`, the release callback would run
on this goroutine and block on `h.mu` forever. A conforming producer is
unlikely to do that, since it calls `release` after the end of the stream or an
error. Could you either correct the spec claim, or list this case as a known
deadlock next to the lock-ordering case the description already mentions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]