[ 
https://issues.apache.org/jira/browse/SPARK-61454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142968#comment-18142968
 ] 

Jungtaek Lim commented on SPARK-61454:
--------------------------------------

[~xuf]
Could you please duplicate the doc over your personal google account or so? I 
don't feel like you can proceed with requiring community members to request 
access to SPIP doc. SPIP doc should be public. 

If you want to avoid in-line comment and instead have a long discussion thread 
in dev@, you can upload a PDF version of the doc instead, though traditionally 
we deal with google doc with personal account.

> SPIP: Extend Real-Time Mode to Distributed Batch SQL and DataFrame Queries
> --------------------------------------------------------------------------
>
>                 Key: SPARK-61454
>                 URL: https://issues.apache.org/jira/browse/SPARK-61454
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 4.0.5
>            Reporter: Ferdinand Xu
>            Priority: Major
>
> {{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution 
> infrastructure to eligible distributed batch SQL and DataFrame queries. }}
> {{The goal is to reduce query latency and improve completed-query throughput 
> on a fixed cluster for interactive and agentic workloads, such as retrieval, 
> feature generation, ranking, and analytical tool calls, while preserving 
> exact Spark SQL results and bounded resource use.}}
> {{The proposed execution path allows downstream stages to consume 
> intermediate data as it arrives across eligible shuffle boundaries, 
> overlapping producer computation, cross-executor transfer, and consumer 
> computation while avoiding unnecessary shuffle materialization.}}
> {{The proposal builds on RTM's existing connected-stage execution and 
> distributed transport, adding batch-specific plan selection, consumer-first 
> bounded admission, aggregate buffer accounting, and execution-attempt 
> lifecycle management.}}
> {{This is an opt-in execution path, disabled by default. Materialized shuffle 
> remains the default, and unsupported plans and exchanges retain ordinary 
> batch execution. The initial scope targets shorter, read-only queries for 
> which restarting the whole query after a transient failure is acceptable; AQE 
> integration and automatic group retry are follow-up work.}}
> {{Early prototype results include a 1.64x speedup on distributed TPC-H 
> scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These 
> results are workload-specific; fixed-cluster concurrent-session latency and 
> throughput validation remain part of the proposed qualification work.}}
> {{{}SPIP 
> doc:{}}}[https://docs.google.com/document/d/1nl56B2ZrSBZBnJCSpkVomnX6Tsrfb5ti7Fat7NlsCqA/edit]
> {{(Please request access since we can't make it public.) }}



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to