Agreed with Ayan. Essentially an Edge node is a physical host or VM that is used by the application to run the job. The users or service users start the process from the Edge node. Edge nodes are added to the cluster for example DEV/TEST/UAT etc.
Edge node normally has all compatible binaries in this case $KAFKA_HOME installation to run Spark jobs. These binaries are normally installed independently (open source) or through parcels say CDH when the cluster admin adds the node to the cluster. I have not seen anyone allowed to directly log in to datanodes, namenodes or kafka nodes (if they are on different Hardware than datanodes etc) to kijck off sparksubmit. it is all done through edge or network servers. For example, you may have 8 Spark processes are running on 8 datanodes say standalone mode one master and multiple worker processes say two on each datanode. However, no workere process will be running on edge node. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 6 June 2017 at 22:42, ayan guha <guha.a...@gmail.com> wrote: > They are all same thing. Essentially it means a machine which is not part > of the cluster but Has all clients. > > On Wed, 7 Jun 2017 at 5:48 am, Irving Duran <irving.du...@gmail.com> > wrote: > >> Where in the documentation did you find "edge node"? Spark would call it >> worker or executor, but not "edge node". Her is some info about yarn logs >> -> https://spark.apache.org/docs/latest/running-on-yarn.html. >> >> >> Thank You, >> >> Irving Duran >> >> On Tue, Jun 6, 2017 at 11:48 AM, Ashok Kumar <ashok34...@yahoo.com> >> wrote: >> >>> Just Straight Spark please. >>> >>> Also if I run a spark job using Python or Scala using Yarn where the log >>> files are kept in the edge node? Are these under logs directory for yarn? >>> >>> thanks >>> >>> >>> On Tuesday, 6 June 2017, 14:11, Irving Duran <irving.du...@gmail.com> >>> wrote: >>> >>> >>> Ashok, >>> Are you working with straight spark or referring to GraphX? >>> >>> >>> Thank You, >>> >>> Irving Duran >>> >>> On Mon, Jun 5, 2017 at 3:45 PM, Ashok Kumar < >>> ashok34...@yahoo.com.invalid> wrote: >>> >>> Hi, >>> >>> I am a bit confused between Edge node, Edge server and gateway node in >>> Spark. >>> >>> Do these mean the same thing? >>> >>> How does one set up an Edge node to be used in Spark? Is this different >>> from Edge node for Hadoop please? >>> >>> Thanks >>> >>> ------------------------------ ------------------------------ --------- >>> To unsubscribe e-mail: user-unsubscribe@spark.apache. org >>> <user-unsubscr...@spark.apache.org> >>> >>> >>> >>> >>> >> -- > Best Regards, > Ayan Guha >
