gitmodimo commented on code in PR #44470:
URL: https://github.com/apache/arrow/pull/44470#discussion_r1985029722
##########
cpp/src/arrow/dataset/file_base.cc:
##########
@@ -472,9 +472,12 @@ Status FileSystemDataset::Write(const
FileSystemDatasetWriteOptions& write_optio
WriteNodeOptions write_node_options(write_options);
write_node_options.custom_schema = custom_schema;
+ // preserve existing order in dataset by setting implicit_order=true
+ bool implicit_ordering = write_node_options.write_options.preserve_order;
acero::Declaration plan = acero::Declaration::Sequence({
- {"scan", ScanNodeOptions{dataset, scanner->options()}},
+ {"scan", ScanNodeOptions{dataset, scanner->options(),
+ /*require_sequenced_output=*/false,
implicit_ordering}},
Review Comment:
Not really. Batch indices are assigned only after they are generated in
source node in the order they are generated. They do have additional
information (the `kAugmentedFields`) that allows rebuilding ordering but this
is not enough to calculate real index in out of order fashion since number of
batches in each fragment is unknown. Only way is sort all batches in
`[__fragment_index,__batch_index]` ordering and then assign indices. This is
what `ordered_sink` node does and it is what `require_sequenced_output=true`
does.
Actually `ordered_sink` becomes obsolete when implicit_ordering converts
`kAugmentedFields` into `batch.index`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
