Hi,
(first ever post)
I experimenting with a Cloudera CDH5 cluster with Spark 1.5.0.
Have tried enabling the CSVSink metrics which seems to work to linux
directories such as /tmp.
However, I'm getting errors when trying to send to an HDFS directory.
Is it possible to use HDFS?
Error message from spark-submit:
16/07/14 11:43:39.489 WARN CsvReporter: Error writing to
application_1466718205476_2664.driver.jvm.total.max
java.io.IOException: No such file or directory
Code extract:
val metrics = "hdfs://moonshot-ha-nameservice/user/jab31/metrics/"
val conf = new SparkConf()
.setAppName("John's Evil Experiments")
.set("spark.metrics.conf.*.sink.csv.class",
"org.apache.spark.metrics.sink.CsvSink")
.set("spark.metrics.conf.*.sink.csv.period","1")
.set("spark.metrics.conf.*.sink.csv.unit","seconds")
.set("spark.metrics.conf.*.sink.csv.directory", metrics)
.set("spark.metrics.conf.worker.sink.csv.period","1")
.set("spark.metrics.conf.worker.sink.csv.unit","seconds")
.set("spark.metrics.conf.master.source.jvm.class","org.apache.spark.metrics.source.JvmSource")
.set("spark.metrics.conf.worker.source.jvm.class","org.apache.spark.metrics.source.JvmSource")
.set("spark.metrics.conf.driver.source.jvm.class","org.apache.spark.metrics.source.JvmSource")
.set("spark.metrics.conf.executor.source.jvm.class","org.apache.spark.metrics.source.JvmSource")
HDFS directory:
# hadoop fs -ls hdfs://moonshot-ha-nameservice/user/jab31/metrics/
Found 1 items
-rw-r--r-- 3 jab31 msc 4 2016-07-14 11:42
hdfs://moonshot-ha-nameservice/user/jab31/metrics/test.txt
Regards,
John
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Is-it-possible-to-send-CSVSink-metrics-to-HDFS-tp27335.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
