Re:Upgrading from Spark SQL 3.2 to 3.3 faild

I need to use cast function to surround computed expression then explain the SQL is ok, for example : cast(a.Split_Amt * b.percent / 100 asdecimal(20,8)) as split_amt I don't know why , is there a config property could compatibility with spark3.2 ? At 2023-02-16 13:47:25,

Upgrading from Spark SQL 3.2 to 3.3 faild

hi,all : I have a sql statement wich can be run on spark 3.2.1 but not on spark 3.3.1 . when I try to explain it, will got error with message: org.apache.spark.sql.catalyst.expressions.Literal cannot be cast to org.apache.spark.sql.catalyst.expressions.AnsiCast execute the sql, error stack is

Re: [Spark Structured Streaming] Could we apply new options of readStream/writeStream without stopping spark application (zero downtime)?

Hi, There is some good documentation under here https://docs.databricks.com/structured-streaming/query-recovery.html Under the “recovery after change in structured streaming query” heading that gives good general guidelines on what can be changed in a “pause” of a stream On Thu, 16 Feb 2023

[Spark Structured Streaming] Could we apply new options of readStream/writeStream without stopping spark application (zero downtime)?

*Component*: Spark Structured Streaming *Level*: Advanced *Scenario*: How-to - *Problems Description* I would like to confirm could we directly apply new options of readStream/writeStream without stopping current running spark structured streaming applications? For

Re: ADLS Gen2 adfs sample yaml configuration

here you go. Please update the values for your specific bucket to be used. spark-defaults.com - to make sure event logs go to ADLS spark.eventLog.enabled true spark.eventLog.dir - abfss:// containen...@storageaccount.dfs.core.windows.net/tenant/spark/eventlogs

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

thnks, Mich .. let me check this On Wed, Feb 15, 2023 at 1:42 AM Mich Talebzadeh wrote: > > It may help to check this article of mine > > > Spark on Kubernetes, A Practitioner’s Guide >

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

It may help to check this article of mine Spark on Kubernetes, A Practitioner’s Guide HTH view my Linkedin profile

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

Your submit command spark-submit --master k8s://https://34.74.22.140:7077 --deploy-mode cluster --name pyspark-example --conf spark.kubernetes.container.image=pyspark-example:0.1 --conf spark.kubernetes.file.upload.path=/myexample src/StructuredStream-on-gke.py pay attention to what it says