This is an automated email from the ASF dual-hosted git repository.

bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git


The following commit(s) were added to refs/heads/main by this push:
     new 3306d79a3a Native Spark docs stale "Spark SQL not supported" row in 
the limitations table (#9483)
3306d79a3a is described below

commit 3306d79a3ace60954ad4d8a02bd2cfefc4a7c873
Author: Sergio Ramazzina <[email protected]>
AuthorDate: Sat Nov 14 13:51:03 2026 +0100

    Native Spark docs stale "Spark SQL not supported" row in the limitations 
table (#9483)
---
 .../ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
index b939b16f43..fec32116e9 100644
--- 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
+++ 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
@@ -613,7 +613,7 @@ Examples: sorted *Group By* (use *Memory Group By*), 
Beam-only I/O, barriers (*B
 |*Not supported.* There is no Hop transform that maps to Spark’s structured 
streaming / Kafka source-sink APIs. For Kafka-oriented Beam pipelines, use the 
Beam engines (e.g. Beam Kafka consumer/producer). Hop’s own Kafka transforms 
(if used) are not native-Spark Dataset I/O.
 
 |Spark SQL
-|*Not supported* as a first-class surface. You cannot submit arbitrary Spark 
SQL / `DataFrame` queries from Hop metadata; the engine only builds a Dataset 
graph from the Hop transform graph (plus native handlers for a fixed set of 
transforms).
+|*Supported* through the xref:pipeline/transforms/spark-sql.adoc[Spark SQL] 
transform: it runs a `spark.sql` statement over the incoming Datasets, each 
registered as a temporary view, and requires an explicit output field list. 
Arbitrary `DataFrame` API code cannot be submitted from Hop metadata.
 
 |MLlib / Spark ML
 |*Not supported.* There are no native handlers or transforms for training, 
pipelines, or model inference via Spark MLlib / `spark.ml`. Keep ML workloads 
in Spark jobs outside Hop, or call external services from ordinary Hop 
transforms where that fits.

Reply via email to