Re: security testing on spark ?

If the port 7077 is open for public on your cluster, that's all you need to take over the cluster. You can read a bit about it here https://www.sigmoid.com/securing-apache-spark-cluster/ You can also look at this small exploit I wrote https://www.exploit-db.com/exploits/36562/ Thanks Best

Re: Spark basicOperators

You can pretty much measure it from the Event timeline listed in the driver ui, You can click on jobs/tasks and get the time that it took for each of it from there. Thanks Best Regards On Thu, Dec 17, 2015 at 7:27 AM, sara mustafa wrote: > Hi, > > The class

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) It passes our tests after we registered 6 new classes with Kryo: kryo.register(classOf[org.apache.spark.sql.catalyst.expressions.UnsafeRow]) kryo.register(classOf[Array[org.apache.spark.mllib.tree.model.Split]])

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

For me, mostly the same as before: tests are mostly passing, but I can never get the docker tests to pass. If anyone knows a special profile or package that needs to be enabled, I can try that and/or fix/document it. Just wondering if it's me. I'm on Java 7 + Ubuntu 15.10, with -Pyarn -Phive

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1.  Ran some regression tests on Spark on Yarn (hadoop 2.6 and 2.7). Tom On Wednesday, December 16, 2015 3:32 PM, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Yes that's what I mean. If they're not quite working, let's disable them, but first, we have to rule out that I'm not just missing some requirement. Functionally, it's not worth blocking the release. It seems like bad form to release with tests that always fail for a non-trivial number of users,

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Thanks Sean for sending me the logs offline. Turns out the tests are failing again, for reasons unrelated to Spark. I have filed https://issues.apache.org/jira/browse/SPARK-12426 for that with some details. In the meanwhile, I agree with Sean, these tests should be disabled. And, again, I don't

Is there any way to select columns of Dataset in addition to the combination of `expr` and `as`?

Hi all, I have two questions about selecting columns of Dataset. First, could you tell me know if there is any way to select TypedColumn columns in addition to the combination of `expr` and `as`? Second, how can we alias such a `expr("name").as[String]` Column? I tried to select a column of

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tests the without-hadoop binaries (so didn't run Hive-related tests) with a test batch including standalone / client, yarn / client and cluster, including core, mllib and streaming (flume and kafka). On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tested a number of tests surrounding DataFrames, Datasets, and ML. On Wed, Dec 16, 2015 at 1:32 PM Michael Armbrust wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December