Re: Application dies, Driver keeps on running

Ah interesting, I stopped spark context and System.exit() from driver with supervise ON and that seemed to start app if it gets killed. On Mon, May 15, 2017 at 5:01 PM, map reduced <k3t.gi...@gmail.com> wrote: > Hi, > I was looking at incorrect place for logs, yes I see some er

Re: Application dies, Driver keeps on running

.com > wrote: > So you are using `client` mode. Right? If so, Spark cluster doesn't manage > the driver for you. Did you see any error logs in driver? > > On Mon, May 15, 2017 at 3:01 PM, map reduced <k3t.gi...@gmail.com> wrote: > >> Hi, >> >> Setup:

Application dies, Driver keeps on running

Hi, Setup: Standalone cluster with 32 workers, 1 master I am running a long running streaming spark job (read from Kafka -> process -> send to Http endpoint) which should ideally never stop. I have 2 questions: 1) I have seen some times Driver is still running but application marked as

Master-Worker communication on Standalone cluster issues

Hi, I was wondering on how often does Worker pings Master to check on Master's liveness? Or is it the Master (Resource manager) that pings Workers to check on their liveness and if any workers are dead to spawn ? Or is it both? Some info: Standalone cluster 1 Master - 8core 12Gb 32 workers -

Re: Reprocessing failed jobs in Streaming job

g to do. > > If you proceed after a failure, you need a place to reliably record > the batches that failed for later processing. > > On Wed, Dec 7, 2016 at 1:46 PM, map reduced <k3t.gi...@gmail.com> wrote: > > Hi, > > > > I am trying to solve this problem - in my st

Re: Spark streaming completed batches statistics

urce. Thanks. > > > https://richardstartin.com/ > > > ---------- > *From:* map reduced <k3t.gi...@gmail.com> > *Sent:* 07 December 2016 19:49 > *To:* Richard Startin > *Cc:* user@spark.apache.org > *Subject:* Re: Spark streaming com

Re: Spark streaming completed batches statistics

Have you checked http://spark.apache.org/docs/latest/monitoring.html#rest-api ? KP On Wed, Dec 7, 2016 at 11:43 AM, Richard Startin wrote: > Is there any way to get this information as CSV/JSON? > > > https://docs.databricks.com/_images/CompletedBatches.png > > >

Reprocessing failed jobs in Streaming job

Hi, I am trying to solve this problem - in my streaming flow, every day few jobs fail due to some (say kafka cluster maintenance etc, mostly unavoidable) reasons for few batches and resumes back to success. I want to reprocess those failed jobs programmatically (assume I have a way of getting

Metrics System not recognizing Custom Source/Sink in application jar

Hi, I've added Custom Source and Sink in my application jar and found a way to get a static fixed metrics.properties on Stand-alone cluster nodes. When I want to launch my application, I give the static path - spark.metrics.conf="/fixed-path/to/metrics.properties". Despite my custom source/sink

Re: Spark Metrics: custom source/sink configurations not getting recognized

Can this be listed as an issue on JIRA? On Wed, Sep 7, 2016 at 10:19 AM, map reduced <k3t.gi...@gmail.com> wrote: > Thanks for the reply, I wish it did. We have an internal metrics system > where we need to submit to. I am sure that the ways I've tried work with >

Re: Spark Metrics: custom source/sink configurations not getting recognized

re is > an anomaly in the metrics beyond the norm. > > Hope this helps. > > Cheers, > Ben > > > On Sep 6, 2016, at 9:52 PM, map reduced <k3t.gi...@gmail.com> wrote: > > Hi, anyone has any ideas please? > > On Mon, Sep 5, 2016 at 8:30 PM, map reduced <k3t.gi.

Re: Spark Metrics: custom source/sink configurations not getting recognized

Hi, anyone has any ideas please? On Mon, Sep 5, 2016 at 8:30 PM, map reduced <k3t.gi...@gmail.com> wrote: > Hi, > > I've written my custom metrics source/sink for my Spark streaming app and > I am trying to initialize it from metrics.properties - but that doesn't > work fr

Spark Metrics: custom source/sink configurations not getting recognized

Hi, I've written my custom metrics source/sink for my Spark streaming app and I am trying to initialize it from metrics.properties - but that doesn't work from executors. I don't have control on the machines in Spark cluster, so I can't copy properties file in $SPARK_HOME/conf/ in the cluster. I

Spark job fails when using checkpointing if a class change in the job

Hi, I have my application jar sitting in HDFS which defines long-running Spark Streaming job and I am using checkpoint dir also in HDFS. Every time I have any changes to the job, I go delete that jar and upload a new one. Now if I upload a new jar and delete checkpoint directory it works fine.

Spark Streaming Job Question about retries and failover

Hi, I have simple spark streaming application which reads data from Kafka and then send this data after transformation on a http end point (or another kafka - for this question let's consider http). I am submitting jobs using job-server . I am