Hi experts, I am just get started using spark and scala.
I am confused how to read local files. I run a spark cluster using docker-compose. There are one master and 2 worker nodes. I think this cluster is so-called standalone cluster. I am trying to submit a simple task to this cluster by this command spark-submit --class example.SimpleApp --master spark://localhost:7077 simple-project_2.12-1.0.jar file:///tmp/sharefiles/README.md This is my test result. Case 1: I mount local(my Mac desktop) /tmp/sharefiles to each worker. It works fine. That’s /tmp/sharefiles/README.md should exists both on my local desktop and worker machine. Othere cases all failed to read. Isn’t it? Why my local desktop need have this file? How can I remove this limitation. For My understanding, file:///xxx should only need exist on worker node. Thanks
