SparkContext taking time after adding jars and asking yarn for resources

In my production setup spark is always taking 40 seconds between these steps like a fixed counter is set. In my local lab these steps take exact 1 second. I am not able to find the exact root cause of this behaviour. My Spark application is running on Hortonworks platform in yarn client mode. Can

Re: Is there a way to limit the sql query result size?

Hi Eric, We are also running into the same issue. Are you able to find some suitable solution to this problem Best Regards -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-way-to-limit-the-sql-query-result-size-tp18316p23272.html Sent from

Re: Spark running slow for small hadoop files of 10 mb size

Thanks for the reply. It indeed increased the usage. There was another issue we found, we were broadcasting hadoop configuration by writing a wrapper class over it. But found the proper way in Spark Code sc.broadcast(new SerializableWritable(conf)) -- View this message in context: