Hello,
I am trying to launch Spark connect on Docker Image
❯ docker run -it apache/spark:3.4.1-scala2.12-java11-r-ubuntu /bin/bash
spark@aa0a670f7433:/opt/spark/work-dir$
/opt/spark/sbin/start-connect-server.sh --packages
org.apache.spark:spark-connect_2.12:3.4.1
starting
this link might help
https://stackoverflow.com/questions/46929351/spark-reading-orc-file-in-driver-not-in-executors
Mich Talebzadeh,
Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
I used the following config and the performance has improved a lot.
.config("spark.sql.orc.splits.include.file.footer", true)
I am not able to find the default value of this config anywhere? Can
someone please share what's the default config of this- is it false?
Also just curious what this
