Hi Spark Community, I have a question regarding the support for User-Defined Functions (UDFs) in Spark Connect, specifically when using Kubernetes as the Cluster Manager.
According to the Spark documentation, UDFs are supported by default for the shell and in standalone applications with additional setup requirements. However, it is not clear whether this support extends to scenarios where Kubernetes is used as the Cluster Manager. cf. https://spark.apache.org/docs/latest/spark-connect-overview.html#:~:text=User%2DDefined%20Functions%20(UDFs)%20are%20supported%2C%20by%20default%20for%20the%20shell%20and%20in%20standalone%20applications%20with%20additional%20set%2Dup%20requirements . Could someone please clarify: 1. Are UDFs supported in Spark Connect when using Kubernetes as the Cluster Manager? 2. If they are supported, are there any specific setup requirements or limitations we should be aware of? 3. If UDFs are not currently supported with Kubernetes as the Cluster Manager, are there any plans to include this support in future releases? Your insights and guidance on this matter would be greatly appreciated. Thank you in advance for your help! Best regards, Yasukazau
