Can I use log4j2.xml in my Apache Saprk application

2016-06-22 Thread Charan Adabala
Hi, We are trying to integrate log4j2.xml instead of log4j.properties in Apache Spark application, We integrated log4j2.xml but, the problem is unable to write the worker log of the application and there is no problem for writing driver log. Can any one suggest how to integrate log4j2.xml in

How to reduce the Executor Computing Time.

2016-03-29 Thread Charan Adabala
>From the below image how can we reduce the computing time for the stages, at some stages the Executor Computing Time is less than 1 sec and some are consuming more than 10sec. Can any one help how to reduce the Executor Computing Time. Thanks in Advance...

Continuously INFO JobScheduler:59 - Added jobs for time *** ms, in my Spark Standalone Cluster.

2016-03-29 Thread Charan Adabala
We are working with Spark Standalone Cluster with 8 Cores and 32GB Ram, with 3 nodes cluster with same configuration. Some times streaming batch completed in less than 1sec. some times it takes more than 10 secs at that time below log will appears in console. 2016-03-29 11:35:25,044 INFO