回复：To monitor the executors of a Spark application

In my opinion, It is 1 Driver Fusion Zhu -- 发件人：Dhiman 发送时间：2020年9月27日(星期日) 23:23 收件人：user 主　题：To monitor the executors of a Spark application Who is responsible to monitor the executors of Spark Application ? 1. Driver Node

Re: Query around Spark Checkpoints

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManager.scala You would need to implement CheckpointFileManager by yourself, which is tightly integrated with HDFS (parameters and return types of methods are mostly from

Re: Query around Spark Checkpoints

Hi, As far as I know, it depends on whether you are using spark streaming or structured streaming. In spark streaming you can write your own code to checkpoint. But in case of structured streaming it should be file location. But main question in why do you want to checkpoint in Nosql, as it's

To monitor the executors of a Spark application

Who is responsible to monitor the executors of Spark Application ? 1. Driver Node 2. Worker Node -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped

I am running Apache Spark Core using Scala 2.12.12 on IntelliJ IDEA 2020.2 with Docker 2.3.0.5 I am running Windows 10 build 2004 Can somebody explain me why am I receiving this

Query around Spark Checkpoints

Hi, I had a query around Spark checkpoints - Can I store the checkpoints in NoSQL or Kafka instead of Filesystem ? Regards, Debu