Was providing IP address instead of FQDN. Providing FQDN helped.
Thanks,
From: Nasrulla Khan Haris
Sent: Wednesday, September 16, 2020 4:11 PM
To: dev@spark.apache.org
Subject: Spark-Locality: Hinting Spark location of the executor does not take
effect.
HI Spark developers,
If I want to hint
Even though i provide hint of wn4's ip address
Spark schedules on wn1
2020-09-17 01:21:50,038 DEBUG [dag-scheduler-event-loop]
scheduler.TaskSetManager: Valid locality levels for TaskSet 11.0:
NODE_LOCAL, RACK_LOCAL, ANY
2020-09-17 01:21:50,039 DEBUG [dispatcher-event-loop-0]
Even though i provide hint of wn4's ip address
Spark schedules on wn1
2020-09-17 01:21:50,038 DEBUG [dag-scheduler-event-loop]
scheduler.TaskSetManager: Valid locality levels for TaskSet 11.0:
NODE_LOCAL, RACK_LOCAL, ANY
2020-09-17 01:21:50,039 DEBUG [dispatcher-event-loop-0]
Even though i provide hint of wn4's ip addressSpark schedules on
wn12020-09-17 01:21:50,038 DEBUG [dag-scheduler-event-loop]
scheduler.TaskSetManager: Valid locality levels for TaskSet 11.0:
NODE_LOCAL, RACK_LOCAL, ANY2020-09-17 01:21:50,039 DEBUG
[dispatcher-event-loop-0] cluster.YarnScheduler:
Sending on behalf of a colleague whose mail isn’t reaching the dev list for
some reason
===
HI Spark developers,
If I want to hint spark to use
HI Spark developers,
If I want to hint spark to use particular list of hosts to execute tasks on. I
see that getBlockLocations is used to get the list of hosts from HDFS.
HI Spark developers,
If I want to hint spark to use particular list of hosts to execute tasks on. I
see that getBlockLocations is used to get the list of hosts from HDFS.
