We are using the spark-runner and for *production monitoring*, the most
popular metrics for us have been ingest rates, (batch) processing times,
and memory usage, all of which which we obtain/calculate based on the
metrics provided by the underlying spark engine (e.g.,
totalProcessedRecords, lastCo
pplication ids
which is very useful.
There are more caveats to speak of in the context of metrics and Beam over
Spark (and streaming in particular), so let me know if you find the gory
details interesting.
-Stas
On Thu, Mar 2, 2017 at 5:37 PM Amit Sela wrote:
> +Stas Levin
>
> On Thu
