avantgardnerio commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3604965154
##########
ballista/core/proto/ballista.proto:
##########
@@ -549,17 +568,30 @@ message PollWorkParams {
repeated TaskStatus task_status = 3;
}
+// Pull-based single-task dispatch. One task processes a slice of
+// partitions; `task_index` names the task within the stage,
+// `global_output_partition_ids` gives the concrete global partition ids it
covers. The
+// task's plan is scheduler-side shrink-restricted so its leaves report
+// `slice.len()` partitions; the writer uses `global_output_partition_ids` to
attach
+// global identity to shuffle files (except in cases where the plan itself
+// resets partitioning: the writer walks its child plan to detect
+// SortPreservingMergeExec or RepartitionExec::Hash and picks the right
+// global mapping).
message TaskDefinition {
uint32 task_id = 1;
uint32 task_attempt_num = 2;
string job_id = 3;
uint32 stage_id = 4;
uint32 stage_attempt_num = 5;
- uint32 partition_id = 6;
+ uint32 task_index = 6;
bytes plan = 7;
string session_id = 9;
uint64 launch_time = 10;
repeated KeyValuePair props = 11;
+ // Global partition ids covered by this task, in slice order.
+ repeated uint32 global_output_partition_ids = 12;
Review Comment:
Can you help me understand the problem? IIUC, the only thing the input
partition number was used for was to restrict the execution plan on the
executor. That functionality has now moved to the scheduler, so plans arrive
already restricted. So in the Spark case (max_paritions_per_task=1) , the only
thing that changes is that this plan has a single input partition (and can
still produce multiple outputs). It doesn't need to know it's input number (and
technically not even really it's output numbers aside from file collisions with
other tasks). If there's something I'm missing, I'm happy to add it back, but
it was coming up as dead code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
