Hi everyone,
I’m trying to use the “extension” feature of the Spark Connect CommandPlugin
(Spark 3.4.1).
I created a simple protobuf message `MyMessage` that I want to send from the
connect client-side to the connect server (where I registered my plugin).
The SparkSession class in `spark-connect-client-jvm` provides a method
`execute` that accepts a `com.google.protobuf.Any`, so I packed the MyMessage
object in an Any:
val spark = SparkSession.builder().remote("sc://localhost").build()
val cmd = com.test.MyMessage.newBuilder().setBlubb("hello world").build()
val googleAny = com.google.protobuf.Any.pack(cmd)
spark.execute(googleAny)
This compiles, but during execution I receive a NoSuchMethodError:
java.lang.NoSuchMethodError: 'void
org.apache.spark.sql.SparkSession.execute(com.google.protobuf.Any)'
After looking around for a while I found that
spark-connect-client-jvm_2.12-3.4.1.jar!SparkSession#execute accepts a
`org.sparkproject.connect.client.com.google.protobuf.Any`.
Am I missing something? Is there an additional build step or should I use a
specific plugin?
Thanks,
Stefan
