NFS is a simple option for this kind of usage, yes. But --files is making N copies of the data - you may not want to do that for large data, or for data that you need to mutate.
On Wed, Nov 25, 2020 at 9:16 PM Artemis User <arte...@dtechspace.com> wrote: > Ah, I almost forgot that there is an even easier solution for your > problem, namely to use the --files option in spark-submit. Usage as > follows: > > --files FILES Comma-separated list of files to be placed in the > working > directory of each executor. File paths of > these files > in executors can be accessed via > SparkFiles.get(fileName). > > -- ND > On 11/25/20 9:51 PM, Artemis User wrote: > > This is a typical file sharing problem in Spark. Just setting up HDFS > won't solve the problem unless you make your local machine as part of the > cluster. Spark server doesn't share files with your local machine without > mounting drives to each other. The best/easiest way to share the data > between your local machine and the Spark server machine is to use NFS (as > Spark manual suggests). You can use a common NFS server and mount > /opt/data drive on both local and the server machine, or run NFS on either > machine and mount the /opt/data on the other. Regardless, you have to > ensure that /opt/data on both local and server machine are pointing to the > save physical drive. Also don't forget to relax the read/write permissions > for all on the drive or map the user ID on both machines. > > Using Fuse may be an option on Mac, but NFS is the standard solution for > this type of problem (Mac supports NFS as well). > > -- ND > On 11/25/20 10:34 AM, Ryan Victory wrote: > > A key part of what I'm trying to do involves NOT having to bring the data > "through" the driver in order to get the cluster to work on it (which would > involve a network hop from server to laptop and another from laptop to > server). I'd rather have the data stay on the server and the driver stay on > my laptop if possible, but I'm guessing the Spark APIs/topology wasn't > designed that way. > > What I was hoping for was some way to be able to say val df = > spark.sql("SELECT * FROM parquet.`*local://*/opt/data/transactions.parquet`") > or similar to convince Spark to not move the data. I'd imagine if I used > HDFS, data locality would kick in anyways to prevent the network shuffles > between the driver and the cluster, but even then I wonder (based on what > you guys are saying) if I'm wrong. > > Perhaps I'll just have to modify the workflow to move the JAR to the > server and execute it from there. This isn't ideal but it's better than > nothing. > > -Ryan > > On Wed, Nov 25, 2020 at 9:13 AM Chris Coutinho <chrisbcouti...@gmail.com> > wrote: > >> I'm also curious if this is possible, so while I can't offer a solution >> maybe you could try the following. >> >> The driver and executor nodes need to have access to the same >> (distributed) file system, so you could try to mount the file system to >> your laptop, locally, and then try to submit jobs and/or use the >> spark-shell while connected to the same system. >> >> A quick google search led me to find this article where someone shows how >> to mount an HDFS locally. It appears that Cloudera supports some kind of >> FUSE-based library, which may be useful for your use-case. >> >> https://idata.co.il/2018/10/how-to-connect-hdfs-to-local-filesystem/ >> >> On Wed, 2020-11-25 at 08:51 -0600, Ryan Victory wrote: >> >> Hello! >> >> I have been tearing my hair out trying to solve this problem. Here is my >> setup: >> >> 1. I have Spark running on a server in standalone mode with data on the >> filesystem of the server itself (/opt/data/). >> 2. I have an instance of a Hive Metastore server running (backed by >> MariaDB) on the same server >> 3. I have a laptop where I am developing my spark jobs (Scala) >> >> I have configured Spark to use the metastore and set the warehouse >> directory to be in /opt/data/warehouse/. What I am trying to accomplish are >> a couple of things: >> >> 1. I am trying to submit Spark jobs (via JARs) using spark-submit, but >> have the driver run on my local machine (my laptop). I want the jobs to use >> the data ON THE SERVER and not try to reference it from my local machine. >> If I do something like this: >> >> val df = spark.sql("SELECT * FROM >> parquet.`/opt/data/transactions.parquet`") >> >> I get an error that the path doesn't exist (because it's trying to find >> it on my laptop). If I run the same thing in a spark-shell on the spark >> server itself, there isn't an issue because the driver has access to the >> data. If I submit the job with submit-mode=cluster then it works too >> because the driver is on the cluster. I don't want this, I want to get the >> results on my laptop. >> >> How can I force Spark to read the data from the cluster's filesystem and >> not the driver's? >> >> 2. I have setup a Hive Metastore and created a table (in the spark shell >> on the spark server itself). The data in the warehouse is in the local >> filesystem. When I create a spark application JAR and try to run it from my >> laptop, I get the same problem as #1, namely that it tries to find the >> warehouse directory on my laptop itself. >> >> Am I crazy? Perhaps this isn't a supported way to use Spark? Any help or >> insights are much appreciated! >> >> -Ryan Victory >> >> >>