[
https://issues.apache.org/jira/browse/ARROW-12873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17396011#comment-17396011
]
David Li commented on ARROW-12873:
----------------------------------
To me this is mostly isomorphic to the original proposal, except using Datum
instead of void*. You could imagine attaching the secondary vector<Datum> to
ExecBatch. This also still has the issue that we need to preserve some global
ordering of metadata and/or know which metadata go with which types of kernels,
something that using a map-like interface as originally proposed would at least
avoid.
To your last point, until relatively recently, there was no way to define
"VarArgs with at least N" parameters without jumping through some hoops. But
even with that support, there's still the issue of ordering the output
metadata/parameters between different kinds of kernels.
> [C++][Compute] Support tagging ExecBatches with arbitrary extra information
> ---------------------------------------------------------------------------
>
> Key: ARROW-12873
> URL: https://issues.apache.org/jira/browse/ARROW-12873
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Ben Kietzman
> Priority: Major
>
> Ideally, ExecBatches could be tagged with arbitrary optional objects for
> tracing purposes and to transmit execution hints from one ExecNode to another.
> These should *not* be explicit members like ExecBatch::selection_vector is,
> since they may not originate from the arrow library. For an example within
> the arrow project: {{libarrow_dataset}} will be used to produce ScanNodes and
> a WriteNodes and it's useful to tag scanned batches with their {{Fragment}}
> of origin. However adding {{ExecBatch::fragment}} would result in a cyclic
> dependency.
> To facilitate this tagging capability, we would need a type erased container
> something like
> {code}
> struct AnySet {
> void* Get(tag_t tag);
> void Set(tag_t tag, void* value, FnOnce<void(void*)> destructor);
> };
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
