Re: Spark Job Performance monitoring approaches

Thank you Georg On Thu, Feb 16, 2017 at 12:30 PM, Georg Heiler wrote: > I know of the following tools > https://sites.google.com/site/sparkbigdebug/home https:// > engineering.linkedin.com/blog/2016/04/dr-elephant-open- >

Re: Spark Job Performance monitoring approaches

I know of the following tools https://sites.google.com/site/sparkbigdebug/home https://engineering.linkedin.com/blog/2016/04/dr-elephant-open-source-self-serve-performance-tuning-hadoop-spark https://github.com/SparkMonitor/varOne https://github.com/groupon/sparklint Chetan Khatri

Spark Job Performance monitoring approaches

Hello All, What would be the best approches to monitor Spark Performance, is there any tools for Spark Job Performance monitoring ? Thanks.

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congrats and welcome! On Mon, Feb 13, 2017 at 6:54 PM, Takuya UESHIN wrote: > Thank you very much everyone! > I really look forward to working with you! > > > On Tue, Feb 14, 2017 at 9:47 AM, Yanbo Liang wrote: > >> Congratulations! >> >> On Mon, Feb

Re: File JIRAs for all flaky test failures

The issue was not with a lack of tooling, I used the url you are describing below to drill down to the exact test failure/stack trace, the problem was that my builds would work like a charm locally but fail with these errors on Jenkins, this was the whole challenge in fixing the unit tests, it

Re: File JIRAs for all flaky test failures

A useful tool for investigating test flakiness is my Jenkins Test Explorer service, running at https://spark-tests.appspot.com/ This has some useful timeline views for debugging flaky builds. For instance, at https://spark-tests.appspot.com/jobs/spark-master-test-maven-hadoop-2.6 (may be slow to

Need Help: getting java.lang.OutOfMemory Error : GC overhead limit exceeded (TransportChannelHandler)

Hi , I am running a spark application and getting out of memory errors in yarn nodemanager logs and container get killed. Please find below for the errors details. Has anyone faced with this issue? *Enabled spark dynamic allocation and yarn shuffle* 2017-02-15 14:50:48,047 WARN

Does anyone here run the TheApacheSpark Youtube channel?

I just saw https://www.youtube.com/user/TheApacheSpark and wondered who 'owns' it? if it's a quasi-official channel, can we list it on http://spark.apache.org/community.html but then, how does one add videos? If it's the Spark Summit video account, as it seems to be at the moment, it shouldn't be

Re: File JIRAs for all flaky test failures

it's not an open-file limit -- i have the jenkins workers set up w/a soft file limit of 100k, and a hard limit of 200k. On Wed, Feb 15, 2017 at 12:48 PM, Armin Braun wrote: > I think one thing that is contributing to this a lot too is the general > issue of the tests taking up a

Re: File JIRAs for all flaky test failures

I think one thing that is contributing to this a lot too is the general issue of the tests taking up a lot of file descriptors (10k+ if I run them on a standard Debian machine). There are a few suits that contribute to this in particular like `org.apache.spark.ExecutorAllocationManagerSuite`

Re: File JIRAs for all flaky test failures

I was working on something to address this a while ago https://issues.apache.org/jira/browse/SPARK-9487 but the difficulty in testing locally made things a lot more complicated to fix for each of the unit tests, should we resurface this JIRA again, I would whole heartedly agree with the

Re: Structured Streaming Spark Summit Demo - Databricks people

Just be warned:  the last time I asked a question about a non-working Databricks Keynote Demo from Spark Summit on the forum mentioned here, they deleted my question!  And i’m a major contributor to those forums!! Often times, those on-stage demos don’t actually work until many months after

Re: Structured Streaming Spark Summit Demo - Databricks people

I don't think this is the right place for questions about Databricks. I'm pretty sure they have their own website with a forum for questions about their product. Maybe this? https://forums.databricks.com/ On Wed, Feb 15, 2017 at 2:34 PM Sam Elamin wrote: > Hey folks >

File JIRAs for all flaky test failures

Hi all, I've noticed the Spark tests getting increasingly flaky -- it seems more common than not now that the tests need to be re-run at least once on PRs before they pass. This is both annoying and problematic because it makes it harder to tell when a PR is introducing new flakiness. To try to

Structured Streaming Spark Summit Demo - Databricks people

Hey folks This one is mainly aimed at the databricks folks, I have been trying to replicate the cloudtrail demo Micheal did at Spark Summit. The code for it can be found here

Re: Update Public Documentation - SparkSession instead of SparkContext

Sorry, The context i am referring is for below URL http://spark.apache.org/docs/2.0.1/programming-guide.html On Wed, Feb 15, 2017 at 1:12 PM, Sean Owen wrote: > When asking a question like this, please actually link to what you are > referring to. Some is intended. > > >

Re: Update Public Documentation - SparkSession instead of SparkContext

There is an existing pull request to update it: https://github.com/apache/spark/pull/16856 But it is a little bit tricky. On Wed, Feb 15, 2017 at 7:44 AM, Chetan Khatri wrote: > Hello Spark Dev Team, > > I was working with my team having most of the confusion