How to create combine DAG visualization?
Hi, How can I create combine DAG visualization of pyspark code instead of separate DAGs of jobs and stages? Thanks b.bhavesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-create-combine-DAG-visualization-tp24653.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to keep history of streaming statistics
Hi Himanshu Mehra, Thanks for reply. I am running spark standalone cluster. I have already set the property regarding logging events in history server as you mentioned. I have also started the history server. I am running my code with awaitTermination(). So it never going to completed jobs. However, I can see it under incomplete jobs category in history server UI. The problem is History server is not showing the "streaming" tab (streaming statistics like Input rate, Scheduling Delay, Processing Time etc.), which is available in Spark UI while job is running. For this do I need to configure something? Where these statistics related files are stored, when job is running? Thanks, b.bhavesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-keep-history-of-streaming-statistics-tp24635p24651.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
How to keep history of streaming statistics
Hello, How can I keep history of streaming statistics for completed applications. Where in Spark, the information presented on UI is stored? Thanks, b.bhavesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-keep-history-of-streaming-statistics-tp24635.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
History server is not receiving any event
Hi, I am working on streaming application. I tried to configure history server to persist the events of application in hadoop file system (hdfs). However, it is not logging any events. I am running Apache Spark 1.4.1 (pyspark) under Ubuntu 14.04 with three nodes. Here is my configuration: File - /usr/local/spark/conf/spark-defaults.conf#In all three nodes spark.eventLog.enabled true spark.eventLog.dir hdfs://master-host:port/usr/local/hadoop/spark_log #in master node export SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=hdfs://host:port/usr/local/hadoop/spark_log Can someone give list of steps to configure history server. Thanks and regards, b.bhavesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/History-server-is-not-receiving-any-event-tp24426.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
