As far as I understand even if I could register the custom source, there is
no way to have a cluster-wide variable to pass to it, i.e. the accumulator
can be modified by tasks, but only the driver can read it and the broadcast
value is constant.
So it seems this custom metrics/sinks fuctionality is
Hi Jerry,
I know that way of registering a metrics, but it seems defeat the whole
purpose. I'd like to define a source that is set within the application, for
example number of parsed messages.
If I register it in the metrics.properties, how can I obtain the instance?
(or instances?)
How can I se
I meant custom Sources, sorry.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Executor-metrics-in-spark-application-tp188p10386.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
I'm also pretty interested how to create custom Sinks in Spark. I'm using it
with Ganglia and the normal metrics from JVM source do show up. I tried to
create my own metric based on Issac's code, but does not show up in Ganglia.
Does anyone know where is the problem?
Here's the code snippet:
clas
I have the same problem (Spark 0.9.1-> 1.0.0 and throws error) and I do call
saveAsTextFile. Recompiled for 1.0.0.
org.apache.spark.SparkException: Job aborted due to stage failure: Task
0.0:10 failed 4 times, most recent failure: Exception failure in TID 1616 on
host r3s1n03.bigdata.emea.nsn-net
I tried to use Kryo as a serialiser isn spark streaming, did everything
according to the guide posted on the spark website, i.e. added the following
lines:
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.set("spark.kryo.registrator", "MyKryoRegistrator");
I also a
Hi,
My Spark installations (both 0.9.1 and 1.0.0) starts up extremely slow when
starting a simple Spark Streaming job.
I have to wait 6 (!) minutes at
INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block
manager
stage and another 4 (!) minutes at
INFO util.MetadataCleaner: R