Thank you cloud0fan. That's really helpful.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
It depends on how you deploy Spark. The writer just writes data to your
specified path(HDFS or local path), but the writer is run on executors. If
you deploy Spark with the local mode, i.e. executor and driver are
together, then you will see the output file on the driver node.
If you deploy Spark
According to Stack Overflow (https://stackoverflow.com/q/40786093) it should
be possible to write file to a local path and the result should be available
on the driver node.
However when I try this:
df.write.parquet("file:///some/path")
the data seems to be written on each node, not a
