Thanks, Matei. I will give it a try.
Larry On Wed, Dec 17, 2014 at 1:01 PM, Matei Zaharia <matei.zaha...@gmail.com> wrote: > > I see, you may have something else configured weirdly then. You should > look at CPU and disk utilization while your Spark job is reading from NFS > and, if you see high CPU use, run jstack to see where the process is > spending time. Also make sure Spark's local work directories > (spark.local.dir) are not on NFS. They shouldn't be though, that should be > /tmp. > > Matei > > On Dec 17, 2014, at 11:56 AM, Larry Liu <larryli...@gmail.com> wrote: > > Hi, Matei > > Thanks for your response. > > I tried to copy the file (1G) from NFS and took 10 seconds. The NFS mount > is a LAN environment and the NFS server is running on the same server that > Spark is running on. So basically I mount the NFS on the same bare metal > machine. > > Larry > > On Wed, Dec 17, 2014 at 11:42 AM, Matei Zaharia <matei.zaha...@gmail.com> > wrote: >> >> The problem is very likely NFS, not Spark. What kind of network is it >> mounted over? You can also test the performance of your NFS by copying a >> file from it to a local disk or to /dev/null and seeing how many bytes per >> second it can copy. >> >> Matei >> >> > On Dec 17, 2014, at 9:38 AM, Larryliu <larryli...@gmail.com> wrote: >> > >> > A wordcounting job for about 1G text file takes 1 hour while input from >> a NFS >> > mount. The same job took 30 seconds while input from local file system. >> > >> > Is there any tuning required for a NFS mount input? >> > >> > Thanks >> > >> > Larry >> > >> > >> > >> > -- >> > View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/wordcount-job-slow-while-input-from-NFS-mount-tp20747.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com >> . >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > For additional commands, e-mail: user-h...@spark.apache.org >> > >> >> >
