Hey Jack,
Emr serverless is a great fit for this. You can get these metrics for each
job when they are completed. Besides that, if you create separate "emr
applications" per group and tag them appropriately, you can use the cost
explorer to see the amount of resources being used.
If emr
It could be simpler and faster to use tagging of resources for billing:
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-tags-billing.html
That could also include other resources (eg s3).
> Am 12.12.2023 um 04:47 schrieb Jack Wells :
>
>
> Hello Spark experts - I’m running
Hello Spark experts - I’m running Spark jobs in cluster mode using a
dedicated cluster for each job. Is there a way to see how much compute time
each job takes via Spark APIs, metrics, etc.? In case it makes a
difference, I’m using AWS EMR - I’d ultimately like to be able to say this
job costs $X
