andygrove commented on issue #2652: URL: https://github.com/apache/datafusion-ballista/issues/2652#issuecomment-6440513882
Answers to the open questions, as implemented in #2657 and recorded in its contributors guide: 1. **Setting name and values.** Keep `ballista.execution.mode = blocking | pipelined`. "Pipelined" is the term Trino and the literature use, and an enum leaves room for a later mode, such as one that pipelines only some stages. 2. **Reserve the widest stage's vcores, or whole executors?** Reserve the widest stage's vcores on each executor. Stages waiting on each other use little CPU, and at TPC-H SF1000 on two executors CPU was not the bottleneck. Reserving whole executors would allow only one pipelined job per executor, which costs too much for short queries. Memory is the real constraint, since every stage's state is resident at once, and the job's memory pool is sized by this reservation. 3. **Wait, or fall back to blocking?** Wait. A silent fallback would change a job's failure semantics (retries) and its performance depending on cluster load. An opt-in fallback, such as a wait timeout after which the job runs in blocking mode, can be added later if people want it. 4. **Eager or lazy producers?** Eager. Bounded queues already stop a producer from running far ahead of its consumer, and a lazy start would add a round trip before every stage. A consumer that never arrives is handled by `ballista.pipelined.registration_timeout_seconds`. Happy to revisit any of these in review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
