andygrove commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3604845759
##########
ballista/core/proto/ballista.proto:
##########
@@ -549,17 +568,30 @@ message PollWorkParams {
repeated TaskStatus task_status = 3;
}
+// Pull-based single-task dispatch. One task processes a slice of
+// partitions; `task_index` names the task within the stage,
+// `global_output_partition_ids` gives the concrete global partition ids it
covers. The
+// task's plan is scheduler-side shrink-restricted so its leaves report
+// `slice.len()` partitions; the writer uses `global_output_partition_ids` to
attach
+// global identity to shuffle files (except in cases where the plan itself
+// resets partitioning: the writer walks its child plan to detect
+// SortPreservingMergeExec or RepartitionExec::Hash and picks the right
+// global mapping).
message TaskDefinition {
uint32 task_id = 1;
uint32 task_attempt_num = 2;
string job_id = 3;
uint32 stage_id = 4;
uint32 stage_attempt_num = 5;
- uint32 partition_id = 6;
+ uint32 task_index = 6;
bytes plan = 7;
string session_id = 9;
uint64 launch_time = 10;
repeated KeyValuePair props = 11;
+ // Global partition ids covered by this task, in slice order.
+ repeated uint32 global_output_partition_ids = 12;
Review Comment:
It isn't clear to me from the current comments how the Spark persona case is
handled, where each task processes a single input partition and potentially
produces multiple output partitions. Where would the input partition id be
specified, now that `partition_id` has been removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
