Hi, we're planning to add a basic Java-API very soon, possibly this week.
There's a ticket for it here:
https://github.com/datastax/cassandra-driver-spark/issues/11
We're open to any ideas. Just let us know what you need the API to have in
the comments.
Regards,
Piotr Kołaczkowski
SparkContext is not serializable and can't be just "sent across" ;)
2014-06-21 14:14 GMT+02:00 Mayur Rustagi :
> You can terminate job group from spark context, Youll have to send across
> the spark context to your task.
> On 21 Jun 2014 01:09, "Piotr Kołaczkow
If the task detects unrecoverable error, i.e. an error that we can't expect
to fix by retrying nor moving the task to another node, how to stop the job
/ prevent Spark from retrying it?
def process(taskContext: TaskContext, data: Iterator[T]) {
...
if (unrecoverableError) {
??? // ter
We are having the same problem. We're running Spark 0.9.1 in standalone
mode and on some heavy jobs workers become unresponsive and marked by
master as dead, even though the worker process is still running. Then they
never join the cluster again and cluster becomes essentially unusable until
we res
No exceptions in any logs. No errors in stdout or stderr.
2014-05-22 11:21 GMT+02:00 Andrew Or :
> You should always call sc.stop(), so it cleans up state and does not fill
> up your disk over time. The strange behavior you observe is mostly benign,
> as it only occurs after you have supposedly
Hi,
Another problem we observed that on a very heavily loaded cluster, if the
worker fails to respond to the heartbeat within 60 seconds, it gets
disconnected permanently from the master and never connects back again. It
is very easy to reproduce - just setup a spark standalone cluster on a
single
Hi,
We observed strange behabiour of Spark 0.9.0 when using sc.stop().
We have a bunch of applications that perform some jobs and then issue
sc.stop() at the end of main. Most of the time, everything works as
desired, but sometimes the applications get marked as "FAILED" by the
master and all rem
