Re: Kill Spark Streaming JOB from Spark UI or Yarn

[Matei Zaharia]

The batches should all have the same application ID, so use that one. You can 
also find the application in the YARN UI to terminate it from there.

Matei

> On Aug 27, 2017, at 10:27 AM, KhajaAsmath Mohammed  
> wrote:
> 
> Hi,
> 
> I am new to spark streaming and not able to find an option to kill it after 
> starting spark streaming context.
> 
> Streaming Tab doesnt have option to kill it.
> 
> Jobs tab too doesn't have option to kill it
> 
> 
> 
> if scheduled on yarn, how to kill that if spark submit is running in 
> background as I will not have an option to find yarn application id. does 
> batches have separate yarn application id or same one?
> 
> Thanks,
> Asmath


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

HTTP 500 if try to access Spark UI in yarn-cluster (only)

[Sebastian YEPES FERNANDEZ]

​Hello,

I am wondering if anyone else is also facing this ​issue:
https://issues.apache.org/jira/browse/SPARK-11147


​

Re: spark ui on yarn

[Koert Kuipers]

my yarn environment does have less memory for the executors.

i am checking if the RDDs are cached by calling sc.getRDDStorageInfo, which
shows an RDD as fully cached in memory, yet it does not show up in the UI


On Sun, Jul 13, 2014 at 1:49 AM, Matei Zaharia matei.zaha...@gmail.com
wrote:

 The UI code is the same in both, but one possibility is that your
 executors were given less memory on YARN. Can you check that? Or otherwise,
 how do you know that some RDDs were cached?

 Matei

 On Jul 12, 2014, at 4:12 PM, Koert Kuipers ko...@tresata.com wrote:

 hey shuo,
 so far all stage links work fine for me.

 i did some more testing, and it seems kind of random what shows up on the
 gui and what does not. some partially cached RDDs make it to the GUI, while
 some fully cached ones do not. I have not been able to detect a pattern.

 is the codebase for the gui different in standalone than in yarn-client
 mode?


 On Sat, Jul 12, 2014 at 3:34 AM, Shuo Xiang shuoxiang...@gmail.com
 wrote:

 Hi Koert,
   Just curious did you find any information like CANNOT FIND ADDRESS
 after clicking into some stage? I've seen similar problems due to lost of
 executors.

 Best,



 On Fri, Jul 11, 2014 at 4:42 PM, Koert Kuipers ko...@tresata.com wrote:

 I just tested a long lived application (that we normally run in
 standalone mode) on yarn in client mode.

 it looks to me like cached rdds are missing in the storage tap of the ui.

 accessing the rdd storage information via the spark context shows rdds
 as fully cached but they are missing on storage page.

 spark 1.0.0

Re: spark ui on yarn

[Shuo Xiang]

Hi Koert,
  Just curious did you find any information like CANNOT FIND ADDRESS
after clicking into some stage? I've seen similar problems due to lost of
executors.

Best,



On Fri, Jul 11, 2014 at 4:42 PM, Koert Kuipers ko...@tresata.com wrote:

 I just tested a long lived application (that we normally run in standalone
 mode) on yarn in client mode.

 it looks to me like cached rdds are missing in the storage tap of the ui.

 accessing the rdd storage information via the spark context shows rdds as
 fully cached but they are missing on storage page.

 spark 1.0.0

Re: spark ui on yarn

[Koert Kuipers]

hey shuo,
so far all stage links work fine for me.

i did some more testing, and it seems kind of random what shows up on the
gui and what does not. some partially cached RDDs make it to the GUI, while
some fully cached ones do not. I have not been able to detect a pattern.

is the codebase for the gui different in standalone than in yarn-client
mode?


On Sat, Jul 12, 2014 at 3:34 AM, Shuo Xiang shuoxiang...@gmail.com wrote:

 Hi Koert,
   Just curious did you find any information like CANNOT FIND ADDRESS
 after clicking into some stage? I've seen similar problems due to lost of
 executors.

 Best,



 On Fri, Jul 11, 2014 at 4:42 PM, Koert Kuipers ko...@tresata.com wrote:

 I just tested a long lived application (that we normally run in
 standalone mode) on yarn in client mode.

 it looks to me like cached rdds are missing in the storage tap of the ui.

 accessing the rdd storage information via the spark context shows rdds as
 fully cached but they are missing on storage page.

 spark 1.0.0

Re: spark ui on yarn

[Matei Zaharia]

The UI code is the same in both, but one possibility is that your executors 
were given less memory on YARN. Can you check that? Or otherwise, how do you 
know that some RDDs were cached?

Matei

On Jul 12, 2014, at 4:12 PM, Koert Kuipers ko...@tresata.com wrote:

 hey shuo,
 so far all stage links work fine for me.
 
 i did some more testing, and it seems kind of random what shows up on the gui 
 and what does not. some partially cached RDDs make it to the GUI, while some 
 fully cached ones do not. I have not been able to detect a pattern.
 
 is the codebase for the gui different in standalone than in yarn-client mode? 
 
 
 On Sat, Jul 12, 2014 at 3:34 AM, Shuo Xiang shuoxiang...@gmail.com wrote:
 Hi Koert,
   Just curious did you find any information like CANNOT FIND ADDRESS after 
 clicking into some stage? I've seen similar problems due to lost of executors.
 
 Best,
 
 
 
 On Fri, Jul 11, 2014 at 4:42 PM, Koert Kuipers ko...@tresata.com wrote:
 I just tested a long lived application (that we normally run in standalone 
 mode) on yarn in client mode.
 
 it looks to me like cached rdds are missing in the storage tap of the ui.
 
 accessing the rdd storage information via the spark context shows rdds as 
 fully cached but they are missing on storage page.
 
 spark 1.0.0
 
 
 

spark ui on yarn

[Koert Kuipers]

I just tested a long lived application (that we normally run in standalone
mode) on yarn in client mode.

it looks to me like cached rdds are missing in the storage tap of the ui.

accessing the rdd storage information via the spark context shows rdds as
fully cached but they are missing on storage page.

spark 1.0.0