Re: Spark on yarn - application hangs

sure NP. I meant these topics [image: image.png] Have a look at this article of mine https://www.linkedin.com/pulse/real-time-processing-trade-data-kafka-flume-spark-talebzadeh-ph-d-/ under section Understanding the Spark Application Through Visualization See if it helps HTH Dr Mich

Re: Spark on yarn - application hangs

How can i check what exactly is stagnant? Do you mean on the DAG visualization on Spark UI? Sorry i'm new to spark. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail:

Re: Spark on yarn - application hangs

Hi, Have you checked matrices from Spark UI by any chance? What is stagnant? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Spark on yarn - application hangs

I've built a spark job in which an external program is called through the use of pipe(). Job runs correctly on cluster when the input is a small sample dataset but when the input is a real large dataset it stays on RUNNING state forever. I've tried different ways to tune executor memory, executor