Ferdinand Xu created SPARK-61454:
------------------------------------

             Summary: SPIP: Extend Real-Time Mode to Distributed Batch SQL and 
DataFrame Queries
                 Key: SPARK-61454
                 URL: https://issues.apache.org/jira/browse/SPARK-61454
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 4.0.5
            Reporter: Ferdinand Xu


{{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution 
infrastructure to eligible distributed batch SQL and DataFrame queries.

The goal is to reduce query latency and improve completed-query throughput on a 
fixed cluster for interactive and agentic workloads, such as retrieval, feature 
generation, ranking, and analytical tool calls, while preserving exact Spark 
SQL results and bounded resource use.}}

{{The proposed execution path allows downstream stages to consume intermediate 
data as it arrives across eligible shuffle boundaries, overlapping producer 
computation, cross-executor transfer, and consumer computation while avoiding 
unnecessary shuffle materialization.}}

{{The proposal builds on RTM's existing connected-stage execution and 
distributed transport, adding batch-specific plan selection, consumer-first 
bounded admission, aggregate buffer accounting, and execution-attempt lifecycle 
management.}}

{{This is an opt-in execution path, disabled by default. Materialized shuffle 
remains the default, and unsupported plans and exchanges retain ordinary batch 
execution. The initial scope targets shorter, read-only queries for which 
restarting the whole query after a transient failure is acceptable; AQE 
integration and automatic group retry are follow-up work.}}

{{Early prototype results include a 1.64x speedup on distributed TPC-H 
scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These 
results are workload-specific; fixed-cluster concurrent-session latency and 
throughput validation remain part of the proposed qualification work. }}

{{SPIP doc:
https://docs.google.com/document/d/1nl56B2ZrSBZBnJCSpkVomnX6Tsrfb5ti7Fat7NlsCqA/edit}}

(Please request access since we can't make it public.)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to