[
https://issues.apache.org/jira/browse/SPARK-61454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142968#comment-18142968
]
Jungtaek Lim commented on SPARK-61454:
--------------------------------------
[~xuf]
Could you please duplicate the doc over your personal google account or so? I
don't feel like you can proceed with requiring community members to request
access to SPIP doc. SPIP doc should be public.
If you want to avoid in-line comment and instead have a long discussion thread
in dev@, you can upload a PDF version of the doc instead, though traditionally
we deal with google doc with personal account.
> SPIP: Extend Real-Time Mode to Distributed Batch SQL and DataFrame Queries
> --------------------------------------------------------------------------
>
> Key: SPARK-61454
> URL: https://issues.apache.org/jira/browse/SPARK-61454
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.0.5
> Reporter: Ferdinand Xu
> Priority: Major
>
> {{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution
> infrastructure to eligible distributed batch SQL and DataFrame queries. }}
> {{The goal is to reduce query latency and improve completed-query throughput
> on a fixed cluster for interactive and agentic workloads, such as retrieval,
> feature generation, ranking, and analytical tool calls, while preserving
> exact Spark SQL results and bounded resource use.}}
> {{The proposed execution path allows downstream stages to consume
> intermediate data as it arrives across eligible shuffle boundaries,
> overlapping producer computation, cross-executor transfer, and consumer
> computation while avoiding unnecessary shuffle materialization.}}
> {{The proposal builds on RTM's existing connected-stage execution and
> distributed transport, adding batch-specific plan selection, consumer-first
> bounded admission, aggregate buffer accounting, and execution-attempt
> lifecycle management.}}
> {{This is an opt-in execution path, disabled by default. Materialized shuffle
> remains the default, and unsupported plans and exchanges retain ordinary
> batch execution. The initial scope targets shorter, read-only queries for
> which restarting the whole query after a transient failure is acceptable; AQE
> integration and automatic group retry are follow-up work.}}
> {{Early prototype results include a 1.64x speedup on distributed TPC-H
> scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These
> results are workload-specific; fixed-cluster concurrent-session latency and
> throughput validation remain part of the proposed qualification work.}}
> {{{}SPIP
> doc:{}}}[https://docs.google.com/document/d/1nl56B2ZrSBZBnJCSpkVomnX6Tsrfb5ti7Fat7NlsCqA/edit]
> {{(Please request access since we can't make it public.) }}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]