Hi Team,
I am using structered streaming in pyspark in azure Databricks, in that I
am creating temp_view from dataframe
(df.createOrReplaceTempView('temp_view')) for performing spark sql query
transformation.
In that I am facing the issue that temp_view not found, so that as a
workaround i have created global temp_view to use.
But same when i have tried to create without streaming, i am able to
perform the temp_view.
write_to_final_table =
(spark.readStream.format('delta').option('ignoreChanges',True).table(f"{delta_table_name}")).writeStream.queryName(f"{query_name}").format("org.elasticsearch.spark.sql").trigger(processingTime=f'1
minutes').outputMode("append").foreachBatch(process_micro_batch).option("checkpointLocation",checkpointdirectory_path).option("mergeSchema",
"true").option("failOnDataLoss", "false").start()
def process_micro_batch(micro_batch_df, batchId) :
micro_batch_df.createOrReplaceTempView("temp_view")
df = spark.sql(f"select * from temp_view")
return df
Here, I am getting error, while reading data from temp_view that temp_view
not found error.
I need to perform or create temp_view (*Not global temp_view)based on the
dataframe, and need to perform the spark sql transformation in structered
streaming.
I have few question in my hand?
1. is strucutered streaming and spark.sql will have different spark.context
within same databricks notebook?
2. If i want to create temp_view based on the dataframe and need to perform
the spark sql operation, how can i create the tempview (Not global
tempview, Since global temp view will be available in the cluster level
across all the notebook)?
Thanks & Regards
