[ 
https://issues.apache.org/jira/browse/SPARK-61454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferdinand Xu updated SPARK-61454:
---------------------------------
    Description: 
{\{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution 
infrastructure to eligible distributed batch SQL and DataFrame queries. }}

{{The goal is to reduce query latency and improve completed-query throughput on 
a fixed cluster for interactive and agentic workloads, such as retrieval, 
feature generation, ranking, and analytical tool calls, while preserving exact 
Spark SQL results and bounded resource use.}}

{{The proposed execution path allows downstream stages to consume intermediate 
data as it arrives across eligible shuffle boundaries, overlapping producer 
computation, cross-executor transfer, and consumer computation while avoiding 
unnecessary shuffle materialization.}}

{{The proposal builds on RTM's existing connected-stage execution and 
distributed transport, adding batch-specific plan selection, consumer-first 
bounded admission, aggregate buffer accounting, and execution-attempt lifecycle 
management.}}

{{This is an opt-in execution path, disabled by default. Materialized shuffle 
remains the default, and unsupported plans and exchanges retain ordinary batch 
execution. The initial scope targets shorter, read-only queries for which 
restarting the whole query after a transient failure is acceptable; AQE 
integration and automatic group retry are follow-up work.}}

{{Early prototype results include a 1.64x speedup on distributed TPC-H 
scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These 
results are workload-specific; fixed-cluster concurrent-session latency and 
throughput validation remain part of the proposed qualification work.}}

{{SPIP doc: 
[https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le]
 }}

  was:
{{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution 
infrastructure to eligible distributed batch SQL and DataFrame queries. }}

{{The goal is to reduce query latency and improve completed-query throughput on 
a fixed cluster for interactive and agentic workloads, such as retrieval, 
feature generation, ranking, and analytical tool calls, while preserving exact 
Spark SQL results and bounded resource use.}}

{{The proposed execution path allows downstream stages to consume intermediate 
data as it arrives across eligible shuffle boundaries, overlapping producer 
computation, cross-executor transfer, and consumer computation while avoiding 
unnecessary shuffle materialization.}}

{{The proposal builds on RTM's existing connected-stage execution and 
distributed transport, adding batch-specific plan selection, consumer-first 
bounded admission, aggregate buffer accounting, and execution-attempt lifecycle 
management.}}

{{This is an opt-in execution path, disabled by default. Materialized shuffle 
remains the default, and unsupported plans and exchanges retain ordinary batch 
execution. The initial scope targets shorter, read-only queries for which 
restarting the whole query after a transient failure is acceptable; AQE 
integration and automatic group retry are follow-up work.}}

{{Early prototype results include a 1.64x speedup on distributed TPC-H 
scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These 
results are workload-specific; fixed-cluster concurrent-session latency and 
throughput validation remain part of the proposed qualification work.}}

{{{}SPIP 
doc:{}}}[https://docs.google.com/document/d/1nl56B2ZrSBZBnJCSpkVomnX6Tsrfb5ti7Fat7NlsCqA/edit]

{{(Please request access since we can't make it public.) }}


> SPIP: Extend Real-Time Mode to Distributed Batch SQL and DataFrame Queries
> --------------------------------------------------------------------------
>
>                 Key: SPARK-61454
>                 URL: https://issues.apache.org/jira/browse/SPARK-61454
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 4.0.5
>            Reporter: Ferdinand Xu
>            Priority: Major
>
> {\{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution 
> infrastructure to eligible distributed batch SQL and DataFrame queries. }}
> {{The goal is to reduce query latency and improve completed-query throughput 
> on a fixed cluster for interactive and agentic workloads, such as retrieval, 
> feature generation, ranking, and analytical tool calls, while preserving 
> exact Spark SQL results and bounded resource use.}}
> {{The proposed execution path allows downstream stages to consume 
> intermediate data as it arrives across eligible shuffle boundaries, 
> overlapping producer computation, cross-executor transfer, and consumer 
> computation while avoiding unnecessary shuffle materialization.}}
> {{The proposal builds on RTM's existing connected-stage execution and 
> distributed transport, adding batch-specific plan selection, consumer-first 
> bounded admission, aggregate buffer accounting, and execution-attempt 
> lifecycle management.}}
> {{This is an opt-in execution path, disabled by default. Materialized shuffle 
> remains the default, and unsupported plans and exchanges retain ordinary 
> batch execution. The initial scope targets shorter, read-only queries for 
> which restarting the whole query after a transient failure is acceptable; AQE 
> integration and automatic group retry are follow-up work.}}
> {{Early prototype results include a 1.64x speedup on distributed TPC-H 
> scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These 
> results are workload-specific; fixed-cluster concurrent-session latency and 
> throughput validation remain part of the proposed qualification work.}}
> {{SPIP doc: 
> [https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le]
>  }}



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to