[
https://issues.apache.org/jira/browse/SPARK-56978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-56978:
--------------------------------
Description:
Motivation:
We want to enhance Spark's usability and interactivity for small-data queries,
specifically on laptops. This can make it more useful for individual users and
beginners prototyping.
Proposal:
This SPIP includes three specific categories of performance improvements,
including optimization improvements for single-file scans, an Arrow-based
df.cache reimplementation, and shuffle-free local execution for small queries.
The community has also suggested a couple other ideas in a similar spirit on
the document, and a couple members have volunteered to help with the
implementation.
SPIP Document:
https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/edit?tab=t.0#heading=h.hj76akdx5ul
> SPIP: Faster queries in local laptop mode for Apache Spark
> ----------------------------------------------------------
>
> Key: SPARK-56978
> URL: https://issues.apache.org/jira/browse/SPARK-56978
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: SPIP
>
> Motivation:
> We want to enhance Spark's usability and interactivity for small-data
> queries, specifically on laptops. This can make it more useful for individual
> users and beginners prototyping.
> Proposal:
> This SPIP includes three specific categories of performance improvements,
> including optimization improvements for single-file scans, an Arrow-based
> df.cache reimplementation, and shuffle-free local execution for small
> queries. The community has also suggested a couple other ideas in a similar
> spirit on the document, and a couple members have volunteered to help with
> the implementation.
> SPIP Document:
> https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/edit?tab=t.0#heading=h.hj76akdx5ul
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
