I just noticed that you're the author of the code I linked in my previous
email. :) It's helpful.
When using `foreachPartition` or `mapPartitions`, I noticed I can't ask Spark
to write the data on the disk using `df.write()` but I need to use the iterator
to do so, which means losing the ability of using partitionBy().
Do you know a workaround? Or I'll be forced to partition data manually.
I think I understand why the job crashes when a single executor does:
`df.write()....save()` writes all the partitions in the same time, which fails
if one of them has died.
Is that right?
Thank you.
Samy
On 10/10/2016 04:58 PM, Samy Dindane wrote:
Hi Cody,
I am writing a spark job that reads records from a Kafka topic and writes them
on the file system.
This would be straightforward if it weren't for the custom checkpointing logic
I want to have; Spark's checkpointing doesn't suit us as it doesn't permit code
updates.
The custom checkpointing would be similar to this:
https://github.com/koeninger/kafka-exactly-once/blob/master/src/main/scala/example/TransactionalPerPartition.scala
I am trying to understand how this would work if an executor crashes, so I
tried making one crash manually, but I noticed it kills the whole job instead
of creating another executor to resume the task.
Is that expected? Is there anything wrong with my approach?
Thank you for your time.
On 10/10/2016 04:29 PM, Cody Koeninger wrote:
What is it you're actually trying to accomplish?
On Mon, Oct 10, 2016 at 5:26 AM, Samy Dindane <s...@dindane.com> wrote:
I managed to make a specific executor crash by using
TaskContext.get.partitionId and throwing an exception for a specific
executor.
The issue I have now is that the whole job stops when a single executor
crashes.
Do I need to explicitly tell Spark to start a new executor and keep the
other ones running?
On 10/10/2016 11:19 AM, Samy Dindane wrote:
Hi,
I am writing a streaming job that reads a Kafka topic.
As far as I understand, Spark does a 1:1 mapping between its executors and
Kafka partitions.
In order to correctly implement my checkpoint logic, I'd like to know what
exactly happens when an executors crashes.
Also, is it possible to kill an executor manually for testing purposes?
Thank you.
Samy
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
