We are using spark.Today I see the FunctionCatalog , and I have seen the
source of
spark\sql\core\src\test\scala\org\apache\spark\sql\connector\DataSourceV2FunctionSuite.scala
and have implements the ScalarFunction.But i still not konw how
to register it in sql
Hi,
I'm not sure I understand the question, but if your question is how to
register (plug-in) your own custom FunctionCatalog, it's through
spark.sql.catalog configuration property, e.g.
spark.sql.catalog.catalog-name=com.example.YourCatalogClass
spark.sql.catalog registers a CatalogPlugin that
Hi,
Start with intercepting stage completions using SparkListenerStageCompleted
[1]. That's Spark Core (jobs, stages and tasks).
Go up the execution chain to Spark SQL with SparkListenerSQLExecutionStart
[2] and SparkListenerSQLExecutionEnd [3], and correlate infos.
You may want to look at how w
OK I managed to load the Python zipped file and the run py.file onto s3 for
AWS EKS to work
It is a bit of nightmare compared to the same on Google SDK which is simpler
Anyhow you will require additional jar files to be added to
$SPARK_HOME/jars. These two files will be picked up after you build
Hi,
ATM I see the most used option for a Spark operator is the one provided by
Google: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
Unfortunately, it doesn't seem actively maintained. Are there any plans to
support an official Apache Spark community driven operator?
Hi,
What exactly are you trying to achieve? Spark on GKE works fine and you can
run Datapoc now on GKE
https://www.linkedin.com/pulse/running-google-dataproc-kubernetes-engine-gke-spark-mich/?trackingId=lz12GC5dRFasLiaJm5qDSw%3D%3D
Unless I misunderstood your point.
HTH
Mich Talebzadeh,
Lead S
I'm not running on GKE. I am wondering what's the long term strategy around
a Spark operator. Operators are the de-facto way to run complex
deployments. The Flink community now has an official community led
operator, and I was wondering if there are any similar plans for Spark.
On Fri, Apr 14, 202
HI there,
I'm a user of spark as part of a Data Engineer profile for daily analytical
work. I write a few commands 100s of times a day and I always wonder if
there would be some way to get spark commands alias instead of
rewriting whole syntax all the time. I checked and there seems no *eval *opti