Hi Arpan,
I presume you are interested in what client was doing.
If you have access to the edge node (where spark code is submitted), look
for the following file
${HOME/.spark_history
example
-rw-r--r--. 1 hduser hadoop 111997 Jun 2 2018 .spark_history
just use shell tools (cat, grep etc) to have a look
Or put it in HDFS somewhere
hdfs dfs -put .spark_history /misc/spark_history ## Spark cannot read a
hidden file
#and read it as text file through sparkRDD in spark-shell
scala> val historyRDD = spark.sparkContext.textFile("/misc/spark_history")
historyRDD: org.apache.spark.rdd.RDD[String] = /misc/spark_history
MapPartitionsRDD[11] at textFile at <console>:23
#print it out
historyRDD.collect().foreach(f=>{println(f)})
HTH
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 29 Jan 2021 at 13:49, Arpan Bhandari <arpan3...@gmail.com> wrote:
> Hi ,
>
> Is there a way to track back spark sql after it has been already run i.e.
> query has been already submitted by a person and i have to back trace what
> query actually got submitted.
>
>
> Appreciate any help on this.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
