short-circuit local reads cannot be used

2014-11-21 Thread Daniel Haviv
Hi, Everytime I start the spark-shell I encounter this message: 14/11/18 00:27:43 WARN hdfs.BlockReaderLocal: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. Any idea how to overcome it ? the short-circuit feature is a big perfomance boost I don't want

Re: Short Circuit Local Reads

2014-09-30 Thread Andrew Ash
://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/ Has anyone been using this in production - curious as to if it made a significant difference from a Spark perspective.

Re: Short Circuit Local Reads

2014-09-30 Thread Kay Ousterhout
in newer versions of Hadoop but I haven't verified it. -Kay -- Forwarded message -- From: Andrew Ash and...@andrewash.com Date: Tue, Sep 30, 2014 at 1:33 PM Subject: Re: Short Circuit Local Reads To: Matei Zaharia matei.zaha...@gmail.com Cc: user@spark.apache.org user

Re: Short Circuit Local Reads

2014-09-30 Thread Andrew Ash
verified it. -Kay -- Forwarded message -- From: Andrew Ash and...@andrewash.com Date: Tue, Sep 30, 2014 at 1:33 PM Subject: Re: Short Circuit Local Reads To: Matei Zaharia matei.zaha...@gmail.com Cc: user@spark.apache.org user@spark.apache.org, Gary Malouf malouf.g

Short Circuit Local Reads

2014-09-17 Thread Gary Malouf
Cloudera had a blog post about this in August 2013: http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/ Has anyone been using this in production - curious as to if it made a significant difference from a Spark perspective.

Re: Short Circuit Local Reads

2014-09-17 Thread Matei Zaharia
about this in August 2013:  http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/ Has anyone been using this in production - curious as to if it made a significant difference from a Spark perspective.