Re: Monitoring tools for spark streaming

2015-09-29 Thread Adrian Tanase 
You can also use the REST api introduced in 1.4 – although it’s harder to parse:

  *   jobs from the same batch are not grouped together
  *   You only get total delay, not scheduling delay

From: Hari Shreedharan
Date: Tuesday, September 29, 2015 at 5:27 AM
To: Shixiong Zhu
Cc: Siva, "user@spark.apache.org<mailto:user@spark.apache.org>"
Subject: Re: Monitoring tools for spark streaming

+1. The Streaming UI should give you more than enough information.

Thanks,
Hari



On Mon, Sep 28, 2015 at 9:55 PM, Shixiong Zhu 
<zsxw...@gmail.com<mailto:zsxw...@gmail.com>> wrote:

Which version are you using? Could you take a look at the new Streaming UI in 
1.4.0?


Best Regards,

Shixiong Zhu

2015-09-29 7:52 GMT+08:00 Siva 
<sbhavan...@gmail.com<mailto:sbhavan...@gmail.com>>:
Hi,

Could someone recommend the monitoring tools for spark streaming?

By extending StreamingListener we can dump the delay in processing of batches 
and some alert messages.

But are there any Web UI tools where we can monitor failures, see delays in 
processing, error messages and setup alerts etc.

Thanks

Re: Monitoring tools for spark streaming

2015-09-29 Thread Otis Gospodnetić 
Hi,

There's also SPM for Spark --
http://sematext.com/spm/integrations/spark-monitoring.html

SPM graphs all Spark metrics and gives you alerting, anomaly detection,
etc. and if you ship your Spark and/or other logs to Logsene -
http://sematext.com/logsene - you can correlate metrics, logs, errors,
etc.  I haven't used SPM with Spark after "AppMap" was introduced (see
http://blog.sematext.com/2015/08/06/introducing-appmap/ ) but I imagine it
would be nice to see a map of Spark nodes talking to each other.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Mon, Sep 28, 2015 at 7:52 PM, Siva <sbhavan...@gmail.com> wrote:

> Hi,
>
> Could someone recommend the monitoring tools for spark streaming?
>
> By extending StreamingListener we can dump the delay in processing of
> batches and some alert messages.
>
> But are there any Web UI tools where we can monitor failures, see delays
> in processing, error messages and setup alerts etc.
>
> Thanks
>
>

Monitoring tools for spark streaming

2015-09-28 Thread Siva 
Hi,

Could someone recommend the monitoring tools for spark streaming?

By extending StreamingListener we can dump the delay in processing of
batches and some alert messages.

But are there any Web UI tools where we can monitor failures, see delays in
processing, error messages and setup alerts etc.

Thanks

Re: Monitoring tools for spark streaming

2015-09-28 Thread Shixiong Zhu 
Which version are you using? Could you take a look at the new Streaming UI
in 1.4.0?

Best Regards,
Shixiong Zhu

2015-09-29 7:52 GMT+08:00 Siva <sbhavan...@gmail.com>:

> Hi,
>
> Could someone recommend the monitoring tools for spark streaming?
>
> By extending StreamingListener we can dump the delay in processing of
> batches and some alert messages.
>
> But are there any Web UI tools where we can monitor failures, see delays
> in processing, error messages and setup alerts etc.
>
> Thanks
>
>

Re: Monitoring tools for spark streaming

2015-09-28 Thread Hari Shreedharan 
+1. The Streaming UI should give you more than enough information.




Thanks, Hari

On Mon, Sep 28, 2015 at 9:55 PM, Shixiong Zhu <zsxw...@gmail.com> wrote:

> Which version are you using? Could you take a look at the new Streaming UI
> in 1.4.0?
> Best Regards,
> Shixiong Zhu
> 2015-09-29 7:52 GMT+08:00 Siva <sbhavan...@gmail.com>:
>> Hi,
>>
>> Could someone recommend the monitoring tools for spark streaming?
>>
>> By extending StreamingListener we can dump the delay in processing of
>> batches and some alert messages.
>>
>> But are there any Web UI tools where we can monitor failures, see delays
>> in processing, error messages and setup alerts etc.
>>
>> Thanks
>>
>>