Thanks for the tips!
I think I figured out what might be causing it. It's the checkpointing to
Microsoft Azure Data Lake Storage (ADLS).
When I use "local checkpointing" it works, but when i use fails when there's
a groupBy in the stream. Weirdly it works when there is no groupBy clause in
the stream.
It's able to create the checkpoint location and the base files on ADLS, but
it can't write any commits. Hence, why I see the first batch of records, but
then it crashes on writing the first commit.
Here's what my checkpointing location looks like:
abfss://"+container_name+"@"+storage_account_name+".dfs.core.windows.net/"+file_name
and my SparkSession:
spark = pyspark.sql.SparkSession.builder\
.appName("pyspark-stream-read")\
.master("spark://spark-master:7077")\
.config("spark.executor.memory", "512m")\
.config("spark.jars.packages",
"io.delta:delta-core_2.12:0.7.0,org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1,org.apache.hadoop:hadoop-azure:3.2.1")
\
.config("spark.sql.extensions",
"io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog",
"org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.config("spark.delta.logStore.class",
"org.apache.spark.sql.delta.storage.AzureLogStore") \
.getOrCreate()
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
