Hi all,
I am learning about the performance difference of Spark when performing a
JOIN problem on Serverless (K8S) and Serverful (Traditional server)
environments.
Through experiment, Spark on K8s tends to run slower than Serverful.
Through understanding the architecture, I know that Spark runs
("local[*]")
.appName(appName)
.getOrCreate()
val sc = spark.sparkContext
val sparkListener = new CustomListener()
sc.addSparkListener(sparkListener)
val inputPath = "s3a://data-join/file00"
sparkListener.wordCount(sc, inputPath)
sc.stop()
}
}
Be
't give you that level
> of detail. You'd have to intercept execution events and correlate them. Not
> an easy task yet doable. HTH.
>
> Pozdrawiam,
> Jacek Laskowski
>
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.
