nchammas commented on code in PR #59613:
URL: https://github.com/apache/spark/pull/59613#discussion_r4411107251
##########
docs/spark-connect-server-libs.md:
##########
@@ -137,16 +72,19 @@ A Spark Server Library consists of the following
components, illustrated in Fig.
4. The client package that exposes the Spark Server Library application logic
to the Spark Client
Application, alongside PySpark or the Scala Spark Client.
<p style="text-align: center;">
- <img src="img/extending-spark-connect-labelled.png" title="Figure 2:
Labelled Architecture" alt="Extending Spark
-Connect Diagram - Labelled Steps"/>
-</p>
+ <img
+ src="img/extending-spark-connect-labelled.png"
+ alt="Extending Spark Connect Diagram - Labelled Steps"
+ />
+ <figcaption style="text-align: center;">Figure 2: Labelled
Architecture</figcaption>
+</p>
-#### (1) Spark Connect Protocol Extension
+# Spark Connect Protocol Extension
Review Comment:
Nit: This and the other section headings should be `##`.
##########
docs/spark-connect-server-libs.md:
##########
@@ -122,12 +59,10 @@ exposed to a client, which differs from existing extension
points in Spark such
[SparkSession
extensions](api/java/org/apache/spark/sql/SparkSessionExtensions.html) or
[Spark Plugins](api/java/org/apache/spark/api/plugin/SparkPlugin.html).
-### Getting Started: Extending Spark with Spark Server Libraries
+* This will become a table of contents (this text will be scraped).
+{:toc}
-Spark Connect is available and supports PySpark and Scala
-applications. We will walk through how to run an Apache Spark server with Spark
-Connect and connect to it from a client application using the Spark Connect
client
-library.
+# Extending Spark with Spark Server Libraries
Review Comment:
This repeats the title from the YAML page metadata. Do we even need this
line? The section is pretty small.
##########
docs/spark-connect-overview.md:
##########
@@ -70,387 +71,49 @@ that developers need to be aware of when using Spark
Connect:
2. By design, the Spark Connect protocol uses Sparks logical
plans as the abstraction to be able to declaratively describe the
operations to be executed
on the server. Consequently, the Spark Connect protocol does not support
all the
- execution APIs of Spark, most importantly RDDs.
+ execution APIs of Spark, most importantly RDDs. Alternatives for missing
RDD functionality
+ are provided as part of the higher-level DataFrame API.
3. Spark Connect provides a session-based client for its consumers. This means
that the
client does not have access to properties of the cluster that manipulate the
- environment for all connected clients. Most importantly, the client does
not have access
- to the static Spark configuration or the SparkContext.
+ environment for all connected clients. Most importantly, the client cannot
change the
+ static Spark configuration and has no access to the SparkContext.
+
+Apart from these differences the Spark Connect API is code-compatible with
classic Spark
+applications: the DataFrame code they contain does not change. What changes is
around that code:
+how the application is launched and configured, where its dependencies go, and
when a plan is
+analyzed. [Setting up Spark Connect](spark-connect-setup.html) covers starting
a server,
+connecting to it and switching an application between the two modes; the
+[Eager vs Lazy](spark-connect-gotchas.html) page covers the behavioral
differences, starting with
+the deferred analysis of query plans.
# Operational benefits of Spark Connect
With this new architecture, Spark Connect mitigates several multi-tenant
operational issues:
**Stability**: Applications that use too much memory will now only impact their
-own environment as they can run in their own processes. Users can define their
-own dependencies on the client and don't need to worry about potential
conflicts
-with the Spark driver.
+own environment as they can run in their own processes. Dependencies installed
on
+the client do not conflict with the Spark driver's, although anything a UDF
imports
+still has to be available on the server.
**Upgradability**: The Spark driver can now seamlessly be upgraded
independently
of applications, for example to benefit from performance improvements and
security fixes.
This means applications can be forward-compatible, as long as the server-side
RPC
definitions are designed to be backwards compatible.
+**Remote connectivity**: The decoupled architecture allows remote connectivity
to Spark beyond SQL
+and JDBC: any application can now interactively use Spark "as a service".
+
+**Simplicity**: The number of APIs exposed to the user is reduced from 3 to 2.
The Spark Connect
Review Comment:
Not clear what the dropped API is here. Is this a reference to RDDs? If so,
I would just call that out directly as a benefit, though at the same time some
people have lamented the lack of RDD support in Spark Connect.
##########
docs/spark-connect-overview.md:
##########
@@ -17,17 +17,18 @@ license: |
See the License for the specific language governing permissions and
limitations under the License.
---
-**Building client-side Spark applications**
-In Apache Spark 3.4, Spark Connect introduced a decoupled client-server
-architecture that allows remote connectivity to Spark clusters using the
-DataFrame API and unresolved logical plans as the protocol. The separation
-between client and server allows Spark and its open ecosystem to be
-leveraged from everywhere. It can be embedded in modern data applications,
-in IDEs, Notebooks and programming languages.
+Spark Connect is a decoupled client-server architecture for Apache Spark. A
thin client
+library speaks to a Spark cluster over gRPC using the DataFrame API and
unresolved logical
+plans as the protocol, so Spark and its open ecosystem can be used from
anywhere:
+embedded in modern data applications, in IDEs, in notebooks and in programming
languages.
+Spark Connect has been part of Apache Spark since the 3.4 release.
To get started, see [Quickstart: Spark
Connect](api/python/getting_started/quickstart_connect.html).
Review Comment:
Not something to necessarily fix here since it predates this PR, but: It's
confusing to offer this link to the PySpark docs to get started when the guide
here is itself also offering to do the same thing.
##########
docs/spark-connect-server-libs.md:
##########
@@ -137,16 +72,19 @@ A Spark Server Library consists of the following
components, illustrated in Fig.
4. The client package that exposes the Spark Server Library application logic
to the Spark Client
Application, alongside PySpark or the Scala Spark Client.
<p style="text-align: center;">
- <img src="img/extending-spark-connect-labelled.png" title="Figure 2:
Labelled Architecture" alt="Extending Spark
-Connect Diagram - Labelled Steps"/>
-</p>
+ <img
+ src="img/extending-spark-connect-labelled.png"
+ alt="Extending Spark Connect Diagram - Labelled Steps"
+ />
+ <figcaption style="text-align: center;">Figure 2: Labelled
Architecture</figcaption>
Review Comment:
Why have both this and previous image? They are repetitive, no? I would
consolidate them into a single image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
