Re: Spark Configurations

Is there a way to get these set by default in spark-sql shell Thanks, Chirag From: Akhil Das mailto:ak...@sigmoidanalytics.com>> Date: Monday, 29 December 2014 5:53 PM To: Chirag Aggarwal mailto:chirag.aggar...@guavus.com>> Cc: "user@spark.apache.org<mailto:user@spark.apach

Spark Configurations

Hi, It seems that spark-defaults.conf is not read by spark-sql. Is it used only by spark-shell? Thanks, Chirag

create table in yarn-cluster mode vs yarn-client mode

Hi, I have a simple app, where I am trying to create a table. I am able to create the table on running app in yarn-client mode, but not with yarn-cluster mode. Is this some known issue? Has this already been fixed? Please note that I am using spark-1.1 over hadoop-2.4.0 App: - import org.ap

column level encryption/decryption with key management

Hi, There have been some efforts going on in providing column level encryption/decryption on hive tables. https://issues.apache.org/jira/browse/HIVE-7934 Is there any plan to extend the functionality over sparksql also? Thanks, Chirag

Re: Does SparkSQL work with custom defined SerDe?

Did https://issues.apache.org/jira/browse/SPARK-3807 fix the issue seen by you? If yes, then please note that it shall be part of 1.1.1 and 1.2 Chirag From: Chen Song mailto:chen.song...@gmail.com>> Date: Wednesday, 15 October 2014 4:03 AM To: "user@spark.apache.org

Value of SHUFFLE_PARTITIONS

Hi, Currently the number of shuffle partitions is config driven parameter (SHUFFLE_PARTITIONS) . This means that anyone who is running a spark-sql query should first of all analyze that what value of SHUFFLE_PARTITIONS would give the best performance for the query. Shouldn't there be a logic i

SparkSql is slow over yarn

When I run SparkSql over yarn, it runs 2-4 times slower as compared to when its run in local mode. Please note that I have a four node yarn setup. Has anyone else also witnessed the same.