Re: Accessing Secure Hadoop from Mesos cluster

DCOS Spark 1.6.1 supports kerberos. It'll be available in DCOS 1.7, to be released in a couple weeks. On Tue, Apr 12, 2016 at 9:57 PM, Tony Kinsley wrote: > I have been working towards getting some spark streaming jobs to run in > Mesos cluster mode (using docker

Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory

Thanks Imran. I will give it a shot when I have some time. Nezih On Thu, Apr 14, 2016 at 9:25 AM Imran Rashid wrote: > Hi Nezih, > > I just reported a somewhat similar issue, and I have a potential fix -- > SPARK-14560, looks like you are already watching it :). You can

Re: DynamoDB data source questions

Hi Reynold, Thanks for the tips. I made some changes based on your suggestion, and now the table scan happens on executors. https://github.com/traviscrawford/spark-dynamodb/blob/master/src/main/scala/com/github/traviscrawford/spark/dynamodb/DynamoDBRelation.scala sqlContext.sparkContext

Organizing Spark ML example packages

Hey Spark devs I noticed that we now have a large number of examples for ML & MLlib in the examples project - 57 for ML and 67 for MLLIB to be precise. This is bound to get larger as we add features (though I know there are some PRs to clean up duplicated examples). What do you think about

[Streaming] Infinite delay when stopping the context

Hello: I use the stop method in my streaming programs to finish the executions of my experiments. However, despite of getting these messages: / //16/04/14 12:03:39 INFO JobGenerator: Stopping JobGenerator immediately// //16/04/14 12:03:39 INFO RecurringTimer: Stopped timer for JobGenerator

Re: Coding style question (about extra anonymous closure within functional transformations)

Yea I agree with you all. Just let you know, this was anyway fixed in https://github.com/apache/spark/commit/6fc3dc8839eaed673c64ec87af6dfe24f8cebe0c On 14 Apr 2016 5:13 p.m., "Takeshi Yamamuro" wrote: > > The latter is simpler and less-typing, I think. > How about

Re: Coding style question (about extra anonymous closure within functional transformations)

The latter is simpler and less-typing, I think. How about adding this as an example in these style guides? // maropu On Thu, Apr 14, 2016 at 4:35 PM, Mark Hamstra wrote: > I don't believe the Scala compiler understands the difference between your > two examples the

Re: Coding style question (about extra anonymous closure within functional transformations)

I don't believe the Scala compiler understands the difference between your two examples the same way that you do. Looking at a few similar cases, I've only found the bytecode produced to be the same regardless of which style is used. On Wed, Apr 13, 2016 at 7:46 PM, Hyukjin Kwon