Hi Kapil,
Thanks for suggestion. Yes, It worked.
Regards
Sachit
On Tue, 9 Mar 2021, 00:19 Kapil Garg, wrote:
> Hi Sachit,
> What do you mean by "spark is running only 1 executor with 1 task" ?
> Did you submit the spark application with multiple executors but only 1 is
> being used and rest ar
Hi Sachit,
What do you mean by "spark is running only 1 executor with 1 task" ?
Did you submit the spark application with multiple executors but only 1 is
being used and rest are idle ?
If that's the case, then it might happen due to spark.locality.wait setting
which is by default set to 3s. This w
Hi All,
I am using Spark 3.0.1 Structuring streaming with Pyspark.
The problem is spark is running only 1 executor with 1 task. Following is
the summary of what I am doing.
Can anyone help on why my executor is 1 only?
def process_events(event):
fetch_actual_data()
#many more steps
def fetch_a
