Hi everyone, I’m trying to use the “extension” feature of the Spark Connect CommandPlugin (Spark 3.4.1).
I created a simple protobuf message `MyMessage` that I want to send from the connect client-side to the connect server (where I registered my plugin). The SparkSession class in `spark-connect-client-jvm` provides a method `execute` that accepts a `com.google.protobuf.Any`, so I packed the MyMessage object in an Any: val spark = SparkSession.builder().remote("sc://localhost").build() val cmd = com.test.MyMessage.newBuilder().setBlubb("hello world").build() val googleAny = com.google.protobuf.Any.pack(cmd) spark.execute(googleAny) This compiles, but during execution I receive a NoSuchMethodError: java.lang.NoSuchMethodError: 'void org.apache.spark.sql.SparkSession.execute(com.google.protobuf.Any)' After looking around for a while I found that spark-connect-client-jvm_2.12-3.4.1.jar!SparkSession#execute accepts a `org.sparkproject.connect.client.com.google.protobuf.Any`. Am I missing something? Is there an additional build step or should I use a specific plugin? Thanks, Stefan