Removing SSL from Spark's internal communications

Hi all, I am wondering what Community will say about the need of removing SSL in Spark's internal communication (File Server and Broadcast Server). The problems I see are the following: 1. Each user must have his own keystore/truststore to use for his jobs - sharing keystores is obviously

Re: Both Spark AM and Client are trying to delete Staging Directory

eing a problem or just questioning the code? > > I have never seen a situation where there's a failure because of that > part of the current code. > > On Fri, Jan 13, 2017 at 3:24 AM, Rostyslav Sotnychenko > <r.sotnyche...@gmail.com> wrote: > > Hi all! > > I am a bit co

Re: Both Spark AM and Client are trying to delete Staging Directory

even the directory doesn't exist, fs.delete(stagingDirPath, true) won't > cause failure but just return false. > > > Rostyslav Sotnychenko wrote > > Hi all! > > > > I am a bit confused why Spark AM and Client are both trying to delete > > Staging Directory. > > >

Both Spark AM and Client are trying to delete Staging Directory

Hi all! I am a bit confused why Spark AM and Client are both trying to delete Staging Directory. https://github.com/apache/spark/blob/branch-2.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1110

Both Spark AM and Client are trying to delete Staging Directory

Hi all! I am a bit confused why Spark AM and Client are both trying to delete Staging Directory. https://github.com/apache/spark/blob/branch-2.1/yarn/ src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1110 https://github.com/apache/spark/blob/branch-2.1/yarn/

[no subject]

Support for Hive 2.x

Hello! I tried compiling Spark 2.0 with Hive 2.0, but as expected this failed. So I am wondering if there is any talks going on about adding support of Hive 2.x to Spark? I was unable to find any JIRA about this. Thanks, Rostyslav

Hive-on-Spark with access to Hive from Spark jobs

Hello! I have a question regarding Hive and Spark. As far as I know, in order to use Hive-on-Spark one need to compile Spark without Hive profile, but that means that it won't be possible to access Hive from normal Spark jobs. How is community going to address this issue? Making two different

Strange exception while reading Parquet files

com%3E>, second is a on some companies Jira <https://jira.talendforge.org/browse/TBD-3615>. The only current workaround I have is upgrading to Spark 1.4.1 but this isn't a solution. Does anyone knows how to deal with it? Thanks in advance, Rostyslav Sotnychenko --